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FOREWORD 


The  Systems  Integration  &  Command/Control  Technical  Area  of  the  U.  S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  is  concerned  with  human 
information  analysis  and  subsequent  product  utilization  in  intelligence  systems.  The  objective 
is  to  provide  both  technological  advances  in  human/machine  aided  tactical  intelligence 
information  processing  and  the  translation  of  these  advances  in  support  of  intelligence 
systems,  design  decisions,  and  the  formulation  of  doctrine  and  procedures.  The  entire 
research  effort  is  responsive  to  requirements  of  ROTE  Project  201 62101 A754,  "Intelligence 
Information  Processing,"  FY  1974  Work  Program  and  to  special  requirements  of  the  U.  S. 
Army  Combat  Arms  Development  Agency  and  the  Intelligence  Center  and  School. 

Achievement  of  the  required  technology  is  often  inhibited  by  a  lack  of  understanding  of 
fundamental  principles.  Where  this  is  true,  the  requirement  is  to  increase  the  scientific  basis 
underlying  the  state  of  the  art.  The  present  publication  provides  a  review  and  analysis  of  one 
such  area-the  assessment  of  subjective  value  or  worth.  A  functional  assessment  technology  is 
needed  to  support  efforts  such  as  determining  the  value  of  intelligence  data  and  improving 
intelligence  collection  procedures. 

ARI  research  in  this  area  is  conducted  as  an  in-house  research  effort  augmented  by 
contracts  with  organizations  selected  as  having  unique  capabilities  and  facilities  for  research 
in  a  specific  area.  The  present  study  was  conducted  jointly  by  personnel  of  the  Army 
Research  Institute  and  the  Industrial  Engineering  Department  of  the  University  of  Wisconsin 
at  Madison. 
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BRIEF 


Raquirement: 

To  analyza  and  avaluate  methods  for  translating  qualitative  subjactive  impressions  of 
worth  into  consistent,  meaningful,  and  uniformly  understandable  quantitative  evaluations. 
Worth  assessment  is  a  vital  ingredient  of  advanced  intelligence  information  processing  methods. 

Procedure: 

Worth  assessment  involves  the  entire  process  of  identifying,  measuring,  and  combining 
factors  to  create  a  conscious,  articulated  worth  structura  as  a  basis  for  making  dacisions.  The 
mathematical  theory  of  worth  assessment  has  been  well  established  and  a  larga  number  of 
models  and  methods  have  been  devaloped.  This  study  raviaws,  classiflas,  and  presents  in 
usable  form  tha  important  methods  for  measuring  worth. 

Assessment  mathods  wera  classified  according  to  their  major  characteristics-use  of 
probability,  type  of  judgment  required,  numbar  and  typa  of  factors  involved,  type  of  output 
needed.  Four  classes  of  methods  wera  diffarantiated:  tha  ordinal,  which  yialds  an  ordinal 
preferenca  scala;  tha  direct,  which  yialds  numerical  preferences;  the  gamble,  using  probability; 
and  the  multivariata,  for  multiple  or  dependent  factors.  Within  these  four  classas,  savan 
genaral  methods  were  reviawed  in  detail;  a  specific  example  of  each  was  given,  with 
step-by-step  procedures  and  possible  variations,  training  and  equipment  needed,  advantages 
and  disadvantages,  possible  pitfalls,  typa  of  results,  ways  of  chacking  accuracy,  and  sources 
of  furthar  information. 

Findings: 

No  single  method  has  been  found  to  be  the  best  for  every  situation;  in  different 
situations  different  methods  are  optimally  effectiva,  dapending  on  tha  requirements  both  of 
the  problem  and  of  the  decision  maker.  The  assessment  method  chosen  must  meat  tha 
requirements  of  the  specific  situation.  A  summary  table  comparing  a  variety  of  dimensions  of 
the  seven  general  methods  is  presented  as  an  aid  for  making  such  a  choice. 

Utilization  of  Findings: 

Methods  of  assessing  subjective  worth  provide  explicit  procedures  for  incorporating 
military  judgment  into  formal  quantitative  expressions  of  worth.  For  example,  in  establishing 
information  acquisition  prioritias  and  managing  collection  assets,  worth  assassmant  procaduras 
can  provida  consistant  and  aasily  communicated  quantitative  evaluations.  This  data  than  can 
be  usad  with  both  existing  and  advanced  mathods  of  collection  management  to  Improva  the 
responsivaness  of  tha  intelligence  systam. 
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INTRODUCTION 

Subjective  worth  forms  the  basis  for  the  selection  of  future  alterna¬ 
tives  and  the  evaluation  of  the  past  actions.  Each  time  an  automobile 
is  purchased,  a  house  is  bought,  a  course  of  action  taken,  subjective 
worth  is  used  as  a  basis.  It  is  the  intuitive  feeling  by  which  one  dif¬ 
ferentiates  between  a  "good  deal”  and  a  "poor  deal"  when  selecting  an 
automobile,  or  by  which  the  pros  and  dons  of  a  prospective  course  of 
action  are  incorporated  into  a  single  evaluation  of  its  worth. 

Techniques  for  measuring  subjective  worth  have  been  utilized  for  many 
years  and  extend  back  to  the  eighteenth  century.  In  the  last  two  or  three 
decades  there  has  been  an  explosive  growth  in  the  interest  and  research 
concerned  with  the  theory  and  assessment  of  subjective  worth.  This  growth 
has  been  stimulated  by  the  use  of  the  axiomatic  method  in  developing  axi¬ 
oms  of  preference  or  worth  which  guarantee,  in  a  formal  mathematical 
sense,  the  ability  to  assess  subjective  worth  numerically.1  Concurrent 
with  the  development  of  theory  a  number  of  procedures  for  assessing 
subjective  worth  have  been  developed.  Although  many  of  these  procedures 
have  been  developed  directly  from  theories  of  preference  and  worth  (e.g., 
the  gamble  methods),  many  are  direct  extensions  of  classical  psycho¬ 
physical  methods.2*3  With  the  exception  of  a  comprehensive  but  short 
review  by  Fishburn  ,4  there  has  been  no  attempt  to  review  the  many  proce¬ 
dures  now  available  for  assessing  subjective  worth.  This  paper  provides 
such  a  review.  The  review  focuses  on  how  to  measure  worth  rather  than 
on  the  theory  underlying  worth  assessment  or  models  for  combining  and 
synthesizing  component  values  in  a  single  worth  for  an  object  or  event. 

Worth  refers  to  the  subjective  or  intuitive  values  with  which  a  person 
evaluates  an  object  or  course  of  action.  People  appear  to  evaluate  objects 
or  courses  of  action  on  the  basis  of  numerous  criteria,  many  of  which  are 
not  quantifiable  in  objective  terms.  Criteria  such  as  "importance"  or 
undesirability"  are  related  to,  yet  quite  different  from,  the  "real  world 
objective"  attributes  usually  associated  with  the  objects  or  actions  (e.g., 
cost,  speed,  color,  size).  These  subjective  criteria  are  combined  by 


1  Fishburn,  P.  Utility  theory  for  decision  making.  New  York:  Wiley,  1970- 

2  Guilford,  J.  P.  Psychometric  methods.  New  York:  McGraw-Hill,  1954 • 

3  Torgerson,  W.  S.  Theory  and  methods  of  scaling.  New  York:  Wiley,  1958* 

4  Fishburn,  P.  Methods  of  estimating  additive  utilities.  Management 
Science,  1967,  12,  435“455- 


the  decision  maker  into  an  intuitive  value  scale  by  which  the  worth  of  ob¬ 
jects  or  actions  are  evaluated.  This  intuitive  process  is  sometimes  re¬ 
ferred  to  as  military  judgment,  engineering  judgment,  or  experience. 

Worth  assessment  refers  to  the  process  of  transforming  and  measuring  these 
subjective  scale  values  onto  an  objective,  real  world  scale.5 6'5 

Worth  assessment  serves  several  purposes.  First,  in  conjunction  with 
appropriate  models,7  it  can  be  used  to  predict  the  decision  behavior  of 
individuals.  Consumer  demand  for  a  product  will  depend  on  the  consumer's 
value  for  that  product.  A  decision-maker's  demand  for  information  will 
depend  on  the  value  he  places  on  the  information.  Worth  assessment  can 
aid  in  prescribing  decisions.  When  the  number  of  decision  alternatives 
becomes  exceedingly  large  and  cumbersome,  techniques  can  be  used  to  de¬ 
compose  the  problem  into  manageable  portions,  assess  the  worth  of  these 
smaller  problems,  and  then  recombine  these  worths  to  prescribe  that  course 
of  action  which  maximizes  total  worth. 8 

Worth  assessment  is  also  a  method  of  communicating  individual  or  group 
values.  This  can  be  especially  useful  in  communicating  policy  from 
superiors  to  subordinates.  Summers,  Taliaferro  and  Fletcher,9  in  a  study 
involving  subjects  learning  another  person's  (target's)  policy,  "found 
that  information  afforded  by  a  mathematical  analysis  of  the  target's 
policy  was  substantially  more  beneficial  to  the  learner  than  was  infor¬ 
mation  provided  by  the  target  himself."  In  addition  to  this  type  of  in¬ 
dividual  communication,  worth  assessment  can  be  used  when  a  group  worth 
function  is  needed.  Since  people  characteristically  vary  tremendously 
in  their  use  of  superlatives  and  adjectives,  analytical  worth  assessment 
can  provide  a  common  medium  of  communication,  more  homogeneous  than 
verbal  description. 

The  procedures  of  worth  assessment  are  the  focus  of  this  paper.  The 
following  section  provides  a  brief  overview  of  the  worth  assessment  process. 
A  system  for  classifying  the  methods  is  provided  in  the  third  section. 


5  Stevens,  S.  S.  A  metric  for  the  social  consensus.  Science.  1966,  151. 
550  -  541. 

6  Stevens,  S.  S.  Issues  in  psychophysical  measurement.  Psychological 
Review.  1971,  78,  426-450. 

7  E.g.,  Fishburn,  P.  Decision  and  value  theory.  New  York:  Wiley,  1964. 

8  Huber,  G.  Multi-attribute  utility  models:  A  review  of  field  and  field¬ 
like  studies.  Management  Science.  1974*  in  press. 

9  Summers,  D.  A.,  J.  D.  Taliaferro,  and  D.  J.  Fletcher.  Judgment  policy 
and  interpersonal  learning.  Behavioral  Science.  1970,  15.  514-521. 
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In' the  fourth  section,  an  example  problem  is  given  which  will  subsequently 
be  used  to  illustrate  each  of  the  techniques  of  worth  assessment--the 
ordinal,  direct,  gamble,  and  multivariate  methods  which  are  discussed  in 
the  next  four  sections.  The  final  section  is  a  brief  summary.  It  should 
be  noted  that  although  some  of  the  material  presented  in  the  fifth  through 
eighth  sections  is  new,  most  of  the  paper  is  tutorial  in  nature. 


THE  WORTH  ASSESSMENT  PROCESS 

The  assessment  of  worth  involves  the  entire  process  of  identifying, 
measuring,  and  combining  attributes  to  create  a  conscious,  well-defined, 
easily  articulated  worth  structure  which  can  form  the  basis  for  evaluation 
and  for  decision.  The  worth  assessment  process  has  the  following  general 
steps : 


1.  Identify  "worth  to  whom,"  the  individual  or  organization  whose 
subjective  worth  is  to  be  assessed. 

2.  Determine  the  scope  of  the  problem  and  identify  the  objectives, 
purposes,  or  uses  of  the  objects  or  events  whose  worths  are  to  be  assessed. 

3.  Identify  the  set  of  alternatives  or  options  to  be  evaluated. 

4.  Determine  the  relevant  dimensions  of  value  (or  the  attributes) 
of  the  objects  or  events  to  be  assessed. 

5.  Develop  physical  measures  for  each  attribute  or  factor. 

6.  Choose  an  appropriate  technique  for  assessing  the  worth  of  each 
attribute  or  factor. 

7.  Assess  the  value  of  each  alternative  on  each  dimension  of  value. 

8.  Choose  an  assessment  model. 

9.  Evaluate  each  alternative  using  this  model. 

10.  Select  the  "best"  alternative. 

It  is  not  the  intent  to  describe  these  steps  in  detail  or  to  analyze  in 
general  the  decision  process;  rather,  the  intent  is  to  provide  the  context 
within  which  worth  assessment  is  conducted. 
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The  first  step  in  worth  assessment  is  to  determine  whose  value  judg¬ 
ments  are  to  be  measured.  The  outcome  of  an  assessment  will  be  critically 
dependent  upon  that  determination.  Although  an  individual  may  be  asked  to 
make  judgment  from  the  point  of  view  of  another  clearly  defined  person, 
the  result  will  be  inextricably  confounded  with  the  values  of  the  individ¬ 
ual  whose  judgments  are  used.10  For  example,  in  selecting  an  automobile, 
the  worth  assigned  to  different  alternatives  will  depend  upon  whether 
the  subjective  worth  of  a  salesman,  a  mechanic,  or  a  prospective  buyer 
is  assessed.  Worth  measured  in  terms  of  an  organization's  values  is  often 
beset  with  similar  difficulties.  The  judgments  of  different  individuals 
will  reflect  different  points  of  view  and  different  values.  In  large 
organizations  it  may  even  be  difficult  to  identify  the  decision  makers 
whose  values  are  sought. 

The  worth  of  an  object,  activity,  or  alternative  is  related  to  the 
extent  to  which  such  is  perceived  as  satisfying  clearly  stated  objectives. 
An  alternative  may  have  several  objectives.  For  example,  in  selecting  an 
automobile,  subjective  worth  may  be  based  on  a  consideration  of  its  use 
for  business,  pleasure,  and  commutation.  The  purpose  influences  the  re¬ 
sulting  worths.  A  sleek  sports  car  may  have  a  greater  subjective  worth 
than  a  sedate  sedan  when  evaluated  as  a  pleasure  vehicle;  however,  the 
sedan  may  have  a  higher  worth  as  a  business  vehicle.  A  clear  statement 
of  objectives  or  purposes  is  necessary  to  establish  the  basis  for  worth 
assessment.*  11 

Often  the  goals  may  be  uncertain. 12  The  decision  maker  may  be  unable 
to  communicate  his  explicit  goals;  he  may  only  mention  some  of  them,  if 
he  feels  that  publicly  divulging  others  may  involve  ridicule;  or  he  may 
not  be  aware  of  his  goals  or  of  which  goals  are  applicable  in  a  given 
situation.  This  is  a  pervasive  problem.  If  the  criterion  with  respect 
to  which  worth  is  to  be  assessed  is  unclear,  the  results  of  any  assessment 
procedure  will  likewise  be  unclear. 


10  Ginsberg,  A.  Decision  analysis  in  clinical  patient  management  with 
application  to  the  pleural-effusion  syndrome.  Rand  Corp. 

Memo  R-75I-RC/NLM,  July  1971. 

11  Pardee,  F.  S.,  C.  T.  Phillips,  and  K.  Smith.  Measurement  and  evaluation 
of  alternative  regional  transportation  mixes;  Vol  II,  Methodology. 

Rand  Corp.  Report  RM-6324-D0T,  August  1970* 

12  Miller,  J.  R.,  III.  A  systematic  procedure  for  assessing  the  worth  of 
complex  alternatives.  USAF  ESD-TR-67-9O,  November  I967.  (AD  662  001) 
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The  third  step  is  to  identify  the  complete  set  of  alternatives  or 
options  to  be  assessed.  In  most  contexts  these  will  be  obvious.  However, 
care  should  be  taken  that  all  the  feasible  alternatives  are  identified 
and  that  the  alternatives  identified  are  in  fact  feasible. 

The  first  three  steps  in  the  worth  assessment  process  define  the  scope 
of  the  assessment  procedure.  They  answer  the  questions:  Whose  worth? 

For  what  purpose?  Of  what  objects  or  activities?  The  fourth  step  is 
to  identify  the  relevant  attributes  or  dimensions  of  value  of  the  alter¬ 
natives  to  be  assessed,  to  decompose  the  alternatives  into  a  set  of 
critical  dimensions  that  influence  the  worth  of  the  alternatives.  Dimen¬ 
sions  of  worth  are  the  attributes,  factors,  or  features  of  the  plausible 
alternatives  that  are  desired  or  valued  by  the  decision  maker;  they  are 
the  dimensions  of  performance  of  the  alternatives  in  satisfying  the 
objective.  13  It  should  be  noted  that  a  complex  problem  can  be  decomposed 
in  more  than  one  way.  Figure  1  illustrates  one  set  of  attributes  which 
might  be  used  in  assessing  the  worth  of  an  automobile.  The  question  of 
which  of  the  various  possible  decompositions  is  more  natural,  easier  to 
obtain,  and  leads  to  the  best  results  cannot  be  settled  on  a  priori 
ground.  However,  some  desirable  properties  of  a  set  of  attributes  can 
be  identified. 

First,  the  set  should  be  complete  and  exhaustive.  All  of  the  attributes 
relevant  to  the  final  decision  should  be  represented  by  items  on  the  list. 
This  guarantees  that  no  important  dimensions  of  value  are  overlooked  in 
the  assessment  procedure. 

Second,  all  of  the  attributes  should  be  within  the  experience  and  com¬ 
prehension  of  the  decision  maker.  Individuals  are  unable  to  consider  more 
than  a  few  value  dimensions  at  a  time  and  thus  may  ignore  potentially 
relevant  information.  However,  if  the  decomposition  is  too  fine-grained 
or  care  is  not  taken,  the  simpler  dimensions  of  value  may  be  incomprehen¬ 
sible  to  the  decision  maker.  In  assessing  the  worth  of  an  automobile, 
power  as  measured  by  rated  horsepower  would  be  meaningful  to  most  individ¬ 
uals,  whereas  such  attributes  as  brake  horsepower  and  thermal  efficiency 
would  be  incomprehensible. 

Third,  the  dimensions  of  value  should  generally  be  mutually  exclusive 
and  independent.  That  is,  each  attribute  should  be  stated  in  such  a  way 
as  not  to  include  any  other  attribute,  either  in  whole  or  in  part,  and 
the  worth  of  an  attribute  at  any  level  should  not  vary  as  a  function  of 


13  The  terms  attribute  and  factor  are  used  interchangeably  in  this  report 
to  refer  to  a  value  dimension. 
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ATTRIBUTE 


MEASURE 


Cost 

Model  Type 


Power 
Efficiency 
Cost  of  Repairs 
Reliability 

Availability  of  Accessories 
Size 

Serviceability 

Handling 

Workmanship 

Warranty 

Attractiveness 


Dollars 

Two-door  Hardtop 
Two-door  Coupe 
Four-door  Hardtop 
Station  Wagon 
Convertible 

Horsepower 

Miles  per  Gallon 

Relative  Consumer  Index 

Years  Between  Failures  (%) 

Number  Possible 

Square  Feet 

Number  of  Dealers  in  Area 
Consumer  Index 
Subjective  Evaluation 
Coverage 

Subjective  Evaluation 


Figure  1.  Automobile  attributes 
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the  levels  of  the  other  attributes.  This  permits  the  use  of  additive 
assessment  models 14 and  reduces  the  number  of  attributes  to  be  evaluated. 

Note  that  although  additive  assessment  models  are  the  most  commonly  used, 
other  techniques  are  available.15*16*17 

There  are  no  formal  algorithms  to  aid  in  defining  the  set  of  value 
dimensions.  Several  quasi-mathematical  approaches  exist  for  identifying 
a  set  of  dimensions  for  a  psychological  space  (e.g.,  factor  analysis, 
multidimensional  scaling).  However,  these  are  cumbersome  and  time 
consuming;  in  addition,  they  usually  require  some  sort  of  overall  judgment 
of  similarity  or  worth  and  thus  are  not  well  suited  for  the  worth  assess¬ 
ment  problem.  The  identification  of  value  dimensions  is  a  critical  step 
and  one  to  which  considerable  time  and  effort  should  be  devoted  in  any 
worth  assessment  problem.  18 

Once  an  acceptable  list  of  relevant  attributes  has  been  constructed, 
the  fifth  step  in  the  procedure  is  to  develop  a  measure  for  each  attribute 
or  dimension.  The  term  "physical  measure"  is  used  to  refer  to  any  tangible 
reading  or  concrete  observation  that  can  be  extracted  from  the  real  world. 

It  is  any  unit  of  measurement  which  can  adequately  reflect  the  plausible 
range  of  values  of  a  given  attribute.  This  measure  serves  to  link  the 
subjective  worths  of  the  decision  maker  to  the  set  of  alternatives  being 
evaluated.19  Note  that  the  level  of  measurement  may  sometimes  be  only 
nominal  or  the  identity  of  different  values  of  the  attribute,  for  example, 
model  type  of  an  automobile  (Figure  l) . 

It  should  be  noted  that  the  use  of  decomposed  worth  assessment  procedures 
has  been  implicit  in  this  discussion  of  dimensions  of  value--that  is, 


14  Fishburn,  1970*  op.  cit. 

15  Fischer,  G.  W.  Multi-dimensional  value  assessment  for  decision  making. 
University  of  Michigan,  Engineering  Psychology  Laboratory  Technical 
Report  037230-2-T,  June  1972.  (a) 

16  Fishburn,  1970,  op.  cit. 

17  Huber,  1974*  op.  cit. 

18  Cf.  O'Connor,  M.  F.  The  assessment  of  worth  for  multi-attributed 
alternatives:  A  survey.  Unpublished  report.  University  of  Michigan. 

1972. 

19  Miller,  J.  R.  Assessing  alternative  transportation  systems.  Rand 
Corp.  Report  RM5865-DOT,  April  1969.  ( PB  185  167) 
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procedures  in  which  the  worth  of  each  attribute  is  evaluated  or  assessed 
separately  and  then  synthesized  into  a  global  assessment  using  mathemati¬ 
cal  techniques . 20  However,  intuitive  assessment  procedures  can  be  used, 
in  which  the  synthesis  of  attribute  information  into  a  global  worth  value 
is  entirely  subjective.  In  both  mathematical  and  intuitive  synthesis  of 
value  information,  it  is  essential  that  agreement  be  reached  on  the 
dimensions  of  value  and  their  associated  measures  for  the  set  of  alterna¬ 
tives  to  be  evaluated.  This  serves  to  make  explicit  the  bases  of  the 
subjective  worth  judgments. 

The  next  steps  in  worth  assessment  are  to  select  a  technique  of  worth 
assessment  and  to  assess  attribute  worths.  A  summary  of  factors  to  be 
considered  in  selecting  a  method  for  worth  assessment  is  presented  fol¬ 
lowing  the  detailed  explanation  of  each  of  the  methods.  All  of  the 
methods  for  assessing  worth  depend  on  the  decision  maker  being  able  to 
delve  into  his  feelings  and  somehow  order  or  quantify  these  feelings. 
However,  these  subjective  impressions  are  not  likely  to  be  stable  over  time 
and  are  dependent  upon  the  short-term  and  long-term  states  of  the  individual 
decision  maker--his  motivation,  job  stress,  time  pressure,  etc.21  The 
decision  analyst  or  researcher  himself  may  induce  problems  by  exerting 
social  pressure,  consciously  or  unconsciously,22'23  orby  neglecting  attributes 
he  feels  are  not  critical. 

The  eighth  step  in  the  worth  assessment  process  is  the  selection  of  an 
assessment  model.  In  many  cases  this  will  be  an  additive  assessment  model, 
as  it  is  usually  found  that  worth  is  insensitive  to  the  algebraic  form  of 
the  model. 24  However,  other  models  are  available  (e.g.,  multiplicative 
models).  These  assessment  models  generally  synthesize  the  component 
values  into  a  single  worth  for  each  alternative,  but  this  is  not  always 
necessary  or  even  desirable.  Thus,  there  are  several  classes  of  models 
such  as  lexicography  in  which  the  assessment  of  an  alternative  is  based 
directly  on  the  component  values.25-26  The  mathematics  of  assessment  models 


20  Cf.  Fischer,  1972a,  op.  cit.;  Fishburn,  1970  >  op.  cit. 

21  Fishburn,  1964,  op.  cit. 

22  Suchman,  F.  Evaluative  research.  New  York:  Russell  Sage  Foundation, 

1967. 

23  Zaleska,  M.  and  N.  Kogan.  Level  of  risk  selected  by  individuals  and 
groups  when  deciding  for  self  and  for  others.  Sociometry t  1971>  34, 
198-213. 

24  Fischer,  1972a,  op.  cit.. 

25  Cf.  Fishburn,  1970,  op.  cit. 

26  MacCrimmon,  K.  R.  Decision  making  among  multi-attribute  alternatives: 
A  survey  and  consolidated  approach.  Rand  Corp.  Memo  RM-4823-ARPA, 
December  1968. 


-  8  - 


are  well  developed.  The  question  of  when  the  models  will  work  is  a  proce¬ 
dural  one,  and  a  review  of  these  procedures  is  outside  the  scope  of  this 
report.  27 

Evaluation  of  the  worth  of  each  alternative  and  selection  of  the  "best" 
alternative  are  the  final  steps  in  the  worth  assessment  process.  These 
depend  on  the  specific  assessment  model  selected  and  the  objectives  of  the 
assessment.  The  result  is  an  evaluation  of  the  worth  of  alternatives  and 
an  explicit  basis  for  decision. 

To  summarize,  the  steps  in  the  process  of  worth  assessment  are  designed 
to  capture  the  essence  of  a  problem  and  to  attain  some  reasonable  degree  of 
measurability.  The  judgments  asked  of  a  decision  maker  are  designed  to 
reflect  the  context  of  the  real-world  decision.  Hypothetical  value  judg¬ 
ments  will  yield  hypothetical  worth  assessments  and  invalidate  the  entire 
decision-making  process.28-29 

A  final  issue  which  must  be  addressed  is  the  validity  of  worth  assess¬ 
ment.  Worth  judgments  cannot  be  confirmed  by  empirical  test.  There  is 
neither  a  right  nor  a  wrong  worth  function;  worth  exists  in  the  minds  of 
men  to  be  accepted  or  rejected  by  other  men.  The  only  external  evaluation 
of  the  results  of  worth  assessment  is  informed  opinion.  The  internal 
consistency  of  worth  judgments  and  the  accurate  prediction  of  decisions  89 
indicate  that  the  methods  of  worth  assessment  are  descriptive  of  human  per¬ 
formance,  and  consistency  can  be  evaluated  using  the  checking  techniques 
discussed  with  each  method.  However,  there  are  no  criteria  or  standards 
for  evaluating  the  validity  of  the  worth  judgments  themselves. 


27  Cf.  Fischer,  1972a,  op.  cit. ;  Fishburn,  1970,  °p.  cit. ;  Huber,  1974, 
op.  cit. 

28  Fishburn,  1964,  op.  cit. 

29  Slovic,  P.  and  S.  Lichtenstein.  Relative  importance  of  probabilities 

and  payoffs  in  risk  taking.  Journal  of  Experimental  Psychology  Monograph, 
1968,  28  (No.  3,  Part  2). 

30  E.g.  Hurst,  P.  and  S.  Siegel.  Prediction  of  decisions  from  a  higher 
ordered  metric  scale  of  utility.  Journal  of  Experimental  Psychology, 

1956,  52,  138-144. 
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THE  CLASSIFICATION  SYSTEM 


In  order  to  facilitate  a  comparison  of  the  characteristics,  inputs, 
processes,  and  output  of  the  methods,  each  worth  assessment  method  has  been 
classified  on  the  basis  of  the  five  attributes  shown  in  Figure  2.  This 
classification  is  a  modification  of  that  used  by  Fishburn.31 

The  first  attribute  describes  the  extent  to  which  the  decision  maker 
must  understand  the  concept  of  probability.  Methods  which  use  probability 
are  generally  based  on  the  classical  axioms  of  utility.  32  Methods  which 
do  not  use  probability  are  generally  not  based  on  a  set  of  axiomatics  of 
choice,  but  rather  on  extensions  of  the  classical  techniques  of 
psychophysics. 33 

The  second  attribute  describes  the  character  of  the  response  expected 
from  the  decision  maker  to  questions  by  the  analyst.  Preference  judgments 
(e.g.,  "I  prefer  #10  to  #1.")  are  relatively  easy  to  make;  indifference 
judgments  (e.g.,  "I  am  indifferent  between  receiving  #2  and  410.")  are 
slightly  more  demanding;  while  quantitative  judgments  (e.g.,  "Receiving 
two  apples  is  four  times  better  than  receiving  one  orange.")  are  generally 
the  most  difficult.34 

The  third  attribute  describes  the  number  of  factors  involved  in  a 
worth  judgment.  In  this  paper,  a  factor,  attribute,  or  factor  level  refers 
to  a  dimension  or  specific  point  on  a  dimension  of  the  object  or  event 
whose  worth  is  being  assessed.  This  attribute  describes  roughly  how  many 
different  factor  dimensions  the  decision  maker  must  simultaneously  consider 
when  making  a  judgment.  In  general,  multiple  factors  increase  the  complexity 
of  the  worth  assessment  task. 

Attribute  four  simply  denotes  whether  a  factor  can  be  described  as 
discrete  (e.g.,  number  of  rooms  in  a  house)  or  continuous  (velocity  of  an 
airplane).  For  many  factors,  the  distinction  between  discrete  and  contin¬ 
uous  becomes  vague.  For  example,  the  cost  of  a  house  might  be  viewed  as 
either  a  discrete  or  continuous  variable. 

Finally,  the  fifth  attribute  in  the  classification  system  describes 
the  output  or  the  product  of  the  method.  The  output  may  only  be  a  ranking 
of  the  attributes  (e.g.,  two  apples  are  preferred  to  one  orange  which  is 
preferred  to  one  apple) .  Approximated  numerical  worth  describes  those 


31  Fishburn,  1967*  op.  cit. 

32  Von  Neumann,  J.  and  0.  Morgens  tern.  Theory  of  games  and  economic 
behavior.  New  York:  Wiley,  1944. 

33  Torgerson,  1958,  op.  cit. 

34  Cf:  Torgerson,  1958,  op.  cit.;  Guilford,  1954,  op.  cit. 
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A/B/C/D/E 


A:  N:  PROBABILITIES  ARE  NOT  USED 
P:  PROBABILITIES  ARE  USED 
P  1/2:  PROBABILITIES  OF  ONE  HALF  ONLY  ARE  USED 

B:  C:  DEPENDS  UPON  PREFERENCE  JUDGEMENTS 
I:  DEPENDS  UPON  INDIFFERENCE  JUDGEMENTS 
Q:  DEPENDS  UPON  QUANTITATIVE  JUDGEMENTS 


C:  NUMBER  OF  FACTORS  INVOLVED  IN  ANY  ONE  JUDGEMENT 


D:  C:  BEST  WITH  CONTINUOUS  FACTORS 
D:  BEST  WITH  Dl  SCRETE  FACTOR S 

E:  USABLE  WITH  EITHER  CONTINUOUS  OR  DISCRETE  FACTORS 


E:  R:  OUTPUT  IS  RANKING  OF  WORTH 

A:  OUTPUT  IS  APPROXIMATED  NUMERICAL  WORTH 
B:  OUTPUT  IS  BOUNDED  NUMERICAL  WORTH 
N:  OUTPUT  IS  DIRECT  NUMERICAL  WORTH 


Figure  2.  Worth  assessment  classification  system 
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outputs  where  numbers  are  assigned  to  attributes  on  a  somewhat  arbitrary 
basis  which  reflects  at  least  their  preference  ordering.  Equal  intervals 
is  one  such  technique.  In  the  above  example,  for  instance,  although  two 
apples,  one  orange,  and  one  apple  lie  on  some  worth  continuum,  with  equal 
intervals  it  is  assumed  that  they  lie  equidistant  from  each  other  (Figure  3) 
The  numbers  assigned  to  their  respective  worths  wi'll  reflect  this  assumption 
As  shown  in  Figure  3>  the  numbers  10,  20  and  30  might  be  assigned  these 
attributes  as  a  measure  of  their  respective  worths.  Bounded  numerical 
worth  refers  to  the  assignment  of  ranges  to  the  worth  of  an  attribute. 

The  output  is  in  the  form  of  two  numbers,  one  representing  the  lowest  value 
and  the  other  the  highest  value  of  worth. 

ID  20  30 

| - 1 - 1 - -f - ►  Worth 

1  APPLE  1  ORANGE  2  APPLES 

Figure  3.  A  scale  of  worth 


Continuing  the  previous  example,  the  value  of  one  apple  might  be  given  as 
7-16,  one  orange  as  19*22,  and  two  apples  as  22-35*  These  bounds  give 
an  indication  of  the  extreme  points  that  the  worth  of  an  attribute  may  take. 
Lastly,  direct  numerical  worth  refers  to  worth  indices  generated  directly 
from  the  decision  maker's  numerical  judgments.  For  instance,  in  the 
example,  if  the  decision  maker  states  that  one  orange  is  1.5  times  better 
than  one  apple  and  that  two  apples  are  2  times  better  than  one  orange,  and 
if  we  make  the  worth  of  one  apple  equal  to  10,  then  one  orange  will  be  worth 
15  and  two  apples  worth  30 •  a  simple  algebraic  manipulation. 


EXAMPLE  PROBLEM 

Each  of  the  worth  assessment  techniques  will  be  illustrated  with  an 
example,  consistent  from  technique  to  technique.  The  example  takes  the 
form  of  assisting  an  individual  (the  decision  maker)  with  purchasing  an 
automobile.  He  knows  that  he  wants  to  buy  an  automobile,  but  because  of 
the  many  factors  or  attributes  involved,  he  is  unable  to  make  a  choice. 

He  has  come  to  the  worth  assessment  researcher  for  assistance.  The  task 
of  the  researcher  will  be  to  help  the  decision  maker  assess  his  worths  so 
that  the  possible  purchases  can  be  evaluated.  Although  in  many  situations 
the  decision  maker  and  worth  assessment  researcher  will  be  the  same  person, 
the  two  roles  have  been  separated  to  help  clarify  the  two  different  per¬ 
spectives.  The  decision  maker  can  often  assist  himself  by  using  the  same 
techniques . 
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The  important  attributes  involved  in  the  decision,  according  to  the 
decision  maker,  have  been  ascertained,  and  measures  of  each  specified 
(Figure  l) .  For  simplicity,  the  first  six  attributes  will  be  used  in 
assessment  method  examples.  Cost  will  be  assumed  to  be  any  dollar  amount 
between  zero  dollars  and  $4000,  the  upper  limit  having  been  specified  by 
the  decision  maker.  In  this  case,  Cost  may  be  thought  of  as  a  continuous 
factor.  Model  Type,  a  discrete  factor,  can  take  on  any  of  the  five  types 
shown  in  Figure  1.  The  acceptable  range  for  Horsepower  has  been  set  at 
50-350  hp,  and  Miles  Per  Gallon  between  10  and  30  mpg.  These  four  ranges 
have  been  specified  by  the  decision  maker;  an  auto  having  an  attribute 
which  falls  outside  any  one  of  these  limits  will  not  be  considered. 
Relative  Cost  of  Repairs  can  vary  between  arbitrary  indices  of  0  and  100, 
based  upon  data  from  a  national  consumer  organization.  And  finally, 
Reliability  can  vary  between  0  and  100$,  based  on  information  from  the 
consumer  organization.  These  six  attributes  will  be  used  in  describing 
the  assessment  methods. 


METHODS  OF  ORDINAL  WORTH  ASSESSMENT 

Two  methods  of  ordinal  worth  assessment  will  be  discussed- -ranking 
and  equivalence  grouping.  The  term  ordinal  is  used  here  because  the 
results  of  the  methods  are  ordinal  preferences.  That  is,  the  results  show 
the  order  or  direction  of  preference,  as  opposed  to  the  magnitude  of 
preference.  This  is  sufficient  information  for  several  decision  models;35 
however,  in  many  situations  worth  must  be  specified  on  an  interval  scale 
(showing  relative  magnitudes  of  preferences).  In  order  to  use  the  ordinal 
methods  in  these  cases,  the  researcher  and/or  the  decision  maker  must 
make  assumptions  regarding  the  translation  of  the  ordinal  results  to 
an  interval  scale.  The  assumption  of  equal  intervals  is  often  made.  In 
any  case,  the  resulting  numerical  worths  on  an  interval  scale  will  only 
approximate  the  decision  maker's  true  preference  magnitudes.  The  fact 
that  only  approximate  numerical  worth  can  be  generated  is  the  main  short¬ 
coming  of  these  methods. 

The  main  advantage  of  the  ordinal  methods  is  that  only  qualitative 
judgments  are  required  of  the  decision  maker.  Since  this  type  of  judgment 
is  commonly  made,  ordinal  methods  are  generally  intuitively  appealing  to 
decision  makers. 

The  two  methods  can  be  used  with  one  or  more  factors  or  attributes. 

For  instance,  the  methods  could  be  used  to  look  at  a  pair  of  factors, 
say  Model  Type  and  Horsepower.  In  this  case  a  factor  level  would  be  for 
instance  a  "two  hundred  horsepower  convertible."  The  number  of  factors 


35  Cf.  MacCrimmon,  1968,  op.  cit. 
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to  be  assessed  increases  rapidly  as  attributes  are  grouped,  since  all 
possible  combinations  of  factor  levels  must  be  assessed.  However,  if 
there  are  a  small  number  of  alternatives  to  choose  from,  then  grouping 
is  not  prohibitive.  The  ordinal  methods  are  described  using  single 
factors . 


Ranking  Methods 

The  ranking  methods  are  the  simplest  worth  assessment  procedures. 
They  require  the  decision  maker  to  order  the  factors  from  most  preferred 
to  least  preferred.  Ranking  is  often  used  in  conjunction  with  other 
methods  which  give  more  precise  numerical  worth  estimates.  The  method: 

N  -  uses  no  probabilities 

C  -  depends  upon  preference  judgments 

2  -  involves  two  factor  levels  at  a  time 

E  -  is  usable  with  either  continuous  or  discrete  factors 

R(A)  -  provides  a  ranking  of  worth  or  approximated  numerical  worth 


Process  Example.  Ranking  is  a  simple  one-step  process,  but  there 
are  many  variations.  As  an  example,  consider  the  ranking  of  various  models 
when  buying  an  automobile.  The  decision  maker  is  presented  a  list  of  the 
alternative  models  (Figure  4)  and  then  given  the  following  instructions: 

"The  automobile  you  are  to  purchase  will  be  one  of  the  five 
models  shown.  Consider  which  model  you  would  prefer. 

Place  a  number  one  next  to  that  model  which  you  most  prefer, 
and  a  number  two  next  to  that  model  which  you  prefer 
next.  Continue  in  this  manner  until  you  have  placed 
a  number  five  to  the  model  least  preferred." 

These  ranks  or  numbers  can  be  used  directly  in  several  decision  strat¬ 
egies  such  as  lexicography  or  dominance.36  The  ranking  tells  us  that  one 
model  is  preferred  to  another,  but  does  not  tell  us  how  much  more  it  is 
preferred. 

The  five  automobile  models  may  be  thought  of  as  points  on  a  continuum 
of  worth,  with  the  order  of  the  points  known  but  the  distances  between 
them  unknown.  If  the  researcher  is  willing  to  make  some  assumptions,  he 
can  approximate  these  distances  (by  using  equal  interval  assumptions,  for 
instance)  and  use  the  ranking  information  in  a  higher  order  decision 
model.  This  may  be  necessary  when  none  of  the  more  powerful  worth  assessment 
techniques  can  be  used.  Such  an  approximate  worth  might  not  reflect  the 
decision  maker's  true  worth  perspective,  however,  and  should  be  used  with 
caution. 


36  See  MacCrimmon,  1968,  op.  cit.,  for  a  discussion  of  various  decision 
strategies . 
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Two-door  hardtop 


RANK 


Two-door  coupe 
Four-door  hardtop 
Station  wagon 
Convertible 


Figure  4.  Example  list  of  factor  levels  to  be  ranked. 


Variations .  Instead  of  presenting  a  list,  it  is  sometimes  advanta¬ 
geous  to  place  each  alternative  on  a  card  and  have  the  decision  maker  sort 
the  cards,  placing  the  most  preferred  on  the  top  of  the  pile  and  so  on 
until  the  least  preferred  is  the  bottom  card.  This  facilitates  reviewing 
the  rank  order  by  the  decision  maker. 

If  the  number  of  alternatives  is  large  the  decision  maker  can  be  asked 
to  first  divide  the  cards  into  a  row  of  several  piles,  where  all  the  cards 
in  any  one  pile  are  more  preferred  than  any  cards  in  the  other  piles  to 
the  right.  In  other  words,  the  pile  farthest  to  the  right  will  contain 
the  least  preferred  factors  and  the  pile  farthest  to  the  left  will  contain 
the  most  preferred.  The  cards  in  each  pile  are  then  ranked,  and  the  piles 
are  combined  to  form  a  complete  ranking. 

A  natural  correspondence  between  worth  and  factor  level  exists  for  many 
continuous  factors.  For  instance,  higher  cost  is  usually  associated  with 
lower  worth.  In  these  cases  only  the  most  preferred  and  least  preferred 
factor  levels  need  be  found.  Then,  given  any  two  points  the  one  closest 
to  the  most  preferred  level  will  be  preferred  to  the  other.  Worth  can 
then  be  approximated  as  a  function  of  the  factor  levels,  if  the  researcher 
is  willing  to  make  further  assumptions. 
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Pair  comparisons  is  a  popular  technique  for  obtaining  a  rank  order.37 
The  decision  maker  is  presented  with  a  series  of  pairs  of  alternatives  and 
is  asked  to  indicate  which  alternative  is  preferred.  Enough  pairs  are 
presented  to  insure  a  complete  ordering.  Figure  5  illustrates  one  method  of 
pair  comparisons. 


Place  an  'X'  next  to  the  model  in  each 
have,  Consider  only  model  type, 

Station  Wagon _ ,  ,  , 

Two-door  Hardtop _ ,  , 

Convertible  X _ .... 

Four-door  Hardtop  X  , 

Two-door  Hardtop  X  ,  , 


PAIR  THAT  YOU  WOULD  PREFER  TO 

,  ,  ,  Convertible  X 
,  Two-door  Coupe  X 

,  , Two-door  Coupe  _ 

,  ,  Station  Wagon  _ 

Four-door  Hardtop  _ 


Figure  5.  Sample  pair  comparison  work  sheet 


From  the  information  shown  we  can  deduce,  given  no  intransitivities,  that 
the  order  of  preference  from  most  to  least  is  convertible,  two-door  coupe, 
two-door  hardtop,  four-door  hardtop  and  station  wagon.  The  decision  maker 
must  be  transitive  in  his  preferences  to  use  pair  comparisons. 


Characteristics .  Cautions:  Two  problems  might  occur  with  this  method. 
First,  the  decision  maker  may  be  indifferent  between  two  factor  levels  and 
unable  to  state  a  clear  preference.  If  this  happens,  assigning  the  same 
approximate  numerical  worth  to  each  will  avoid  the  problem.  Alternatively, 
the  decision  maker  can  be  forced  to  arbitrarily  assign  one  factor  level  a 
rank  higher  than  the  other.  The  second  problem  concerns  intransitivity 
and  is  most  likely  to  occur  with  pair  comparisons.  For  example,  if  in 


37  Torgerson,  1958,  op.  cit. 


Figure  5  the  first  factor  level  in  each  pair  was  preferred,  then  it  could 
be  inferred  that  a  station  wagon  is  preferred  to  a  four-door  hardtop  and 
at  the  same  time  a  four-door  hardtop  is  preferred  to  a  station  wagon. 

This  type  of  impasse  should  be  discussed  with  the  decision  maker  and 
rectified  by  him. 

Second,  the  reader  should  be  cautioned  to  not  place  too  much  faith 
in  approximate  numerical  worths  obtained  with  this  method.  These  numeri¬ 
cal  values  should  be  used  explicitly  in  a  decision  model  only  if  the  deci¬ 
sion  is  relatively  insensitive  to  small  variations  in  the  worth  of  the 
factor,  as  is  often  the  case.  38 

Equipment:  Depending  upon  the  variation  being  used,  either  paper  and 
pencil  or  cards  are  necessary  with  these  methods. 

Results:  The  output  of  this  method  is  a  ranking  of  the  alternatives 

or  an  approximate  numerical  representation  of  the  rank  order.  Approximate 
numerical  worth  can  be  used  in  numerical  utility  models,  but  with  the 
cautions  previously  mentioned. 


Advantages.  The  method  is  the  easiest  for  the  decision  maker  of  the 
several  we  will  consider.  It  requires  no  training,  little  equipment,  and  is 
intuitively  appealing,  since  rankings  (preference  statements)  are  common 
judgments  for  people  to  make. 


Disadvantages .  The  main  disadvantage  of  the  method  is  that  only  approx¬ 
imate  numerical  worth  can  be  obtained,  and  because  of  this  it  may  not  be 
applicable  in  many  situations.  In  these  cases,  ranking  must  be  augmented 
with  another  method  to  get  more  precise  worth  numbers. 


Checking  Techniques.  After  the  ranking  has  been  established,  the 
decision  maker  can  be  presented  with  several  additional  choice  preference 
situations.  His  preferences  in  each  pair  should  conform  to  the  expected 
preference.  For  instance,  if  we  assume  the  decision  maker  has  given  us  the 
information  in  Figure  5,  he  might  be  asked: 

'*Do  you  prefer  a  station  wagon  or  a  two-door  hardtop?” 

Based  on  the  ranking  developed  from  Figure  5  he  should  be  expected  to 
prefer  the  two-door  hardtop.  Any  discrepancy  found  using  the  additional 
pairs  should  be  cleared  up  with  the  decision  maker. 


38  Von  Winterfeldt,  D.  and  W.  Edwards.  Costs  and  payoffs  in  perceptual 
research.  In  E.  C.  Carterette  and  M.  P.  Friedman  (Eds.),  Handbook 
of  Perception.  New  York:  Academic  Press,  in  press. 
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Sources  of  Information.  The  theory  and  general  discussions  of  the 
method  can  be  found  in  Guilford,39  Thur stone, 40  and  Torgerson.41  The 
basic  method  was  used  by  Gulliksen42  for  determining  the  values  of  various 
meats •  In  his  study,  subjects  were  asked  to  state  preferences  between 
Tongue,  Pork,  Lamb,  Beef,  and  Steak,  as  well  as  between  pairs  of  the 
meats.  For  example,  the  subjects  were  asked  to  choose  the  most  preferred 
in  each  of  the  following  three  pairs: 

1.  Port  -  Beef 

2.  Pork  and  Beef  -  Steak 

3-  Pork  and  Steak  -  Tongue  and  Beef 

It  was  assumed  that  the  subjects  must  eat  both  meats  in  a  composite  alterna¬ 
tive.  The  subjects  rated  all  possible  pairs  except  those  in  which  the 
same  meat  appeared  in  both  of  the  choices.  The  resulting  relations  were 
solved  using  a  least  squares  technique  to  get  scale  values. 

By  looking  at  pairs  of  meats  it  was  found  that  some  meats  seemed  to 
have  positive  worth,  while  others  had  negative  worth.  That  is,  the  value 
of  a  Steak  and  Tongue  composite  was  higher  than  Tongue  alone,  the  value 
of  Steak  and  Lamb  was  greater  than  Lamb  alone,  etc.,  indicating  that  Steak 
added  worth  when  used  in  a  composite.  On  the  other  hand,  the  value  of 
Pork  and  Beef  was  lower  than  Beef  alone,  the  value  of  Pork  and  Lamb  was 
lower  than  Lamb  alone,  etc.,  thus  indicating  that  Pork  detracts  from 
composite  value  and  thus  has  a  negative  value.  This  seems  to  indicate  the 
existence  of  negative  worth. 

Bechtel43-44  used  pair  comparison  to  develop  similarity  ratings  of 
words  and  phrases,  and  describes  several  methods  for  obtaining  approximate 
numerical  worth.  Aumann  and  Kruskal45  used  pair  comparison  to  obtain  an 
analytic  allocation  strategy  for  naval  equipment. 


39  Guilford,  1954,  op.  cit. 

40  Thurstone,  L.  L.  The  measurement  of  values.  Chicago:  University 
of  Chicago  Press,  1959* 

41  Torgerson,  1958;  op.  cit. 

42  Gulliksen,  H.  Measurement  of  subjective  values.  Psychometr ika ,  1956; 
21,  229-244. 

43  Bechtel,  G.  Comparative  scaling  of  unidimensional  discrimination  and 
similarity  data.  Psychometr ika,  1966,  51.  75"84. 

44  Bechtel,  G.  Folded  and  unfolded  scaling  from  preferential  paired  com¬ 
parisons.  Journal  of  Mathematical  Psychology,  1968,  333"357- 

45  Aumann,  R.  J.  and  J.  B.  Kruskal.  Assigning  quantitative  values  to 

qualitative  factors  in  the  naval  electronics  problem.  Naval  Research 
Logistics  Quarterly,  1959;  1-16. 
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Equivalence  Grouping 


This  method  involves  the  classification  by  the  decision  maker  of  the 
worth  of  levels  of  a  factor  into  fixed  categories.  One  factor  level  at 
a  time  is  rated  onto  a  category  rating  scale.  Only  approximate  numerical 
worth  is  possible  with  this  method.  The  method: 

N  -  uses  no  probabilities 

I  -  depends  on  indifference  judgments 

1  -  involves  one  factor  level  at  a  time 

D  -  works  best  with  discrete  factors 

A  -  provides  approximated  numerical  worth 

Process  Example.  Step  1:  Select  all  the  factors  and  factor  levels 
whose  worths  are  to  be  assessed.  As  an  example,  let  us  assume  that  the 
factor  to  be  rated  is  the  model  of  car:  the  states  (or  factor  levels) 
are  two-door  hardtop,  two-door  coupe,  convertible,  station  wagon  and  four- 
door  hardtop. 

Step  2:  Choose  a  rating  scale  such  as  a  Likert  or  Thurstone  scale. 

In  our  example  we  shall  choose  a  seven-point  Likert  scale.  A  discussion 
of  the  various  scales  is  given  below. 

Step  3:  Provide  cues  for  the  scale.  The  words  or  cues  on  a  rating 
scale  serve  two  purposes:  They  help  give  the  decision  maker  a  sense  of 
what  the  scale  represents,  and  they  help  distinguish  between  different 
categories.  The  choice  of  words  used  is  important.  In  general,  the  words 
should  be  clear  to  both  the  researcher  and  decision  maker  and  should  be 
relevant  to  the  situation.  It  is  assumed,  and  the  researcher  must  ensure, 
that  the  words  across  the  scale  are  in  a  natural  rank  order.  For  instance, 
if  the  decision  maker  believes  that  a  ffhighly  desirable”  factor  is  better 
than  a  "most  desirable"  factor  (see  Figure  6),  any  inference  drawn  from 
his  groupings  will  be  invalid,  since  the  equivalence  grouping  assumes  that 
"most  desirable"  is  better  than  "highly  desirable." 

An  alternative  to  naming  all  categories  is  to  name  only  the  extreme 
ones.  This  indicates  the  range  of  possible  responses  but  does  nothing  to 
distinguish  between  adjacent  categories.  Guilford46  discusses  more 
completely  the  issues  involved  in  constructing  such  rating  scales. 


46  Guilford,  1954,  op.  cit. 
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Two-door  Hardtop 


Two-door  Coupe 
Convertible 
Station  Wagon 
Four-door  Hardtop 
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Figure  6.  Sample  questionnaire  for  equivalence  grouping 


Step  4:  Frame  the  factors  for  which  the  worths  are  to  be  assessed  and 
the  rating  scale  in  a  format  like  that  shown  in  Figure  6.  The  decision  maker 
is  then  told: 

"This  questionnaire  is  for  determining  your  preferences 
for  the  different  automobile  models  listed  below.  For 
the  purposes  here,  please  consider  only  your  preference 
for  model  type,  and  disregard  any  other  factors.  Please 
mark  your  preference  for  each  model  on  the  scale  provided." 

Step  5:  Assign  approximate  worth  numbers  to  the  categories.  This 
step  is  necessary  only  if  an  interval  scale  of  worth  is  needed.  The  as¬ 
signment  of  approximate  worth  numbers  depends  upon  the  assumption  of  equal 
intervals.  This  assumption  allows  the  categories  to  be  assigned  consecutive 
integers  representing  their  place  along  the  scale.  These  numbers  are  often 
included  on  the  questionnaire  with  the  cues. 


20 


In  the  example,  we  assumed  equal  intervals  and  assigned  the  numbers 
7,  6,  5>  4,  3,  2,  and  1  to  the  categories  from  left  to  right.  Thus,  each 
factor  level  rated  as  most  desirable  will  be  assigned  a  worth  of  7>  those 
rated  highly  desirable  a  6,  etc.  The  number  assigned  to  a  factor  level 
will  represent  its  approximate  numerical  worth. 


Variations .  The  scale  need  not  have  exactly  seven  intervals.  Five, 
seven,  nine  and  eleven  are  most  often  U9ed,  but  as  few  as  two  and  as  many 
as  twenty-one  intervals  have  been  reported. The  optimal  number  of  intervals 
depends  upon  two  factors.  First,  as  the  number  of  intervals  increases  it 
becomes  increasingly  more  difficult  for  the  decision  maker  to  perceive 
differences  between  categories  and  hence  to  rate  the  factor  levels.  Second, 
more  intervals  allow  a  more  precise  breakdown  of  values,  as  long  as  the 
decision  maker  is  able  to  perceive  accurately  the  differences  between 
categories.  The  discrimination  of  the  decision  maker  and  the  precision 
of  the  resulting  worth  must  be  balanced  according  to  the  particular 
situation. 

A  scale  may  be  replaced  by  containers  labeled  by  the  points  on  the 
scale.  Factor  levels  can  be  placed  on  cards  and  the  cards  sorted  into 
the  appropriate  containers. 


Characteristics .  Cautions:  Although  the  researcher  may  perceive  the 
distances  between  successive  points  to  be  equal,  the  decision  maker  may 
perceive  them  to  be  unequal.  For  example,  he  might  disagree  that  the  differ¬ 
ence  between  Most  Desirable  and  Highly  Desirable  is  the  same  as  the  difference 
between  Indifferent  and  Undesirable#  If  the  researcher  suspects  this  to 
be  the  case,  he  should  ask  the  decision  maker  to  rate  the  relative  worth 
of  the  various  points  on  the  scale  by  another  method. 

Equipment:  Paper  and  pencil  is  required  unless  the  container  varia¬ 

tion  is  used.  When  this  method  is  used  for  the  assessment  of  worth  by  a 
large  number  of  decision  makers,  preprinted  questionnaires  may  be  necessary. 

Results:  The  output  of  this  method  is  numbers  that  approximate  the 
worth  of  given  factor  levels.  The  range  of  the  worth  values  is  arbitrary 
and  assigned  by  the  researcher  based  on  his  needs. 

The  results  are  often  different  from  those  obtained  using  ranking 
methods  because  some  categories  may  contain  several  factor  levels  and  others 
may  contain  none.  Depending  upon  the  number  of  categories,  the  decision 
maker  may  be  able  to  give  more  information  than  in  the  ranking  procedures. 
Specifically,  he  can  give  some  indications  of  worth  magnitude  by  hcrw  close 


47  Guilford,  1954,  op*  cit. 


-  21  - 


or  far  away  two  factors  are  placed.  On  the  other  hand,  when  there  are 
few  categories  he  may  not  be  able  to  give  as  much  information  as  with 
the  ranking  methods,  since  each  category  will  contain  many  factor  levels 
and  rankings  within  categories  are  unknown. 

Information  Needed:  A  precise  statement  of  the  factors  to  be  assessed 
is  the  only  information  needed  for  this  method. 


Advantages .  This  is  one  of  the  easiest  methods  of  assessment  and  is 
widely  used.  It  requires  no  additional  skills  on  the  part  of  the  decision 
maker. 


Disadvantages .  It  is  difficult  to  obtain  very  precise  worth  assess¬ 
ments  by  this  method  since  numerical  worth  values  can  only  be  approximated 
by  an  interval. 


Checking  Techniques.  After  the  factors  have  been  placed  in  equivalence 
groups,  the  decision  maker  can  be  presented  with  choice  preference  situations 
(see  Figure  5)  involving  factor  levels  in  different  groups.  The  preferred 
factor  level  should  be  that  indicated  by  the  equivalence  grouping.  Alter¬ 
nately,  he  can  be  asked  to  rank  order  the  factor  levels,  which  can  then 
be  used  to  check  the  equivalence  grouping.  Finally,  factor  lists  can  contain 
repeated  factor  levels  which  should  all  be  grouped  into  the  same  category. 


Sources  of  Information.  Guilford  48  and  Thurstone49  deal  with  this 
technique.  Vroom  and  Deci,50  in  a  more  theoretical  treatment,  describe  the 
use  of  five  and  eleven  interval  scales  to  measure  importance  of  attributes 
of  jobs. 

Slovic  61  used  the  method  in  a  study  of  stockbrokers*  decision  making. 
The  purpose  was  to  discover  how  two  stockbrokers  used  information  when 
deciding  to  buy  stocks.  The  subjects  rated  the  growth  potential  of  128 
hypothetical  stocks  described  by  11  factors  normally  associated  with  stock, 
such  as  price/earnings  ratio,  past  year*s  performance,  and  volume  trend. 
Each  factor  was  described  by  only  two  levels,  such  as  up  or  down  for  trends, 
and  high  or  low  for  yields.  This  was  done  to  simplify  the  procedure. 


48  Guilford,  1954,  op.  cit. 

49  Thurstone,  1959*  °P*  cit. 

50  Vroom,  V.  H.  and  E.  Deci.  The  stability  of  post-decision  dissonance: 

A  follow-up  study  of  the  job  attitudes  of  business  school  graduates. 
Organization  Behavior  and  Human  Performance,  I97I,  6,  36-49. 

51  Slovic,  P.  Analyzing  the  expert  judge:  A  descriptive  study  of  a  stock¬ 
broker*  s  decision  process.  Journal  of  Applied  Psychology,  1969,  *33. 
255-263. 
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The  stockbrokers  were  presented  with  the  hypothetical  companies  and 
asked  to  make  recommendations  on  a  nine  point  rating  scale.  Categories  1, 

4,  and  9  were  labeled  respectively  nstrong  recommendation  not  to  buy,'1 

’’slight  recommendation  not  to  buy,  ”  ’’neutral,”  ’’slight  recommendation  to 
buy”  and  ’’strong  recommendation  to  buy.”  The  category  numbers  were  used 
directly  in  an  analysis  of  variance  to  determine  the  effects  of  each  factor 
and  factor  pair  on  the  rating  of  the  companies'  stock.  This  procedure  is 
the  reverse  of  that  shown  in  our  example;  Slpvic  analyzed  decisions  to 
find  worth  factors,  and  in  our  example  worth  factors  are  analyzed  to 
determine  a  decision. 

In  other  studies,  Goldberg62  used  an  eleven-point  scale  for  assessing 
diagnostic  judgments,  and  Stagner 63  used  seven  point  scales  for  determining 
corporate  decision  making  practices.  In  a  comparison  of  various  scaling 
techniques,  Patrick,  Bush  and  Chen64  used  an  eleven  point  scale  as  one 
method.  It  was  found  to  correlate  highly  with  other  methods.  They  have 
also  used  this  approach  in  developing  health  status  indices.56  Cooper66 
used  the  method  in  studies  of  preference  for  various  soft  drinks.  An 
extensive  program  of  research  in  the  area  of  food  preference  as  well  as 
methods  for  statistical  inference  is  reviewed  by  Bock  and  Jones.67 


62  Goldberg,  L.  Five  models  of  clinical  judgment:  An  empirical  comparison 

between  linear  and  nonlinear  representations  of  the  human  inference  pro¬ 
cess.  Organizational  Behavior  and  Human  Performance,  1971*  458-479* 

63  Stagner,  R.  Corporate  decision  making:  An  empirical  study.  Journal  of 
Applied  Psychology,  1969>  55.  1-15* 

54  Patrick,  D.,  J.  Bush,  and  M.  Chen.  Comparing  three  methods  for  measuring 
social  preference  for  levels  of  function.  Unpublished  report.  University 
of  California  at  San  Diego,  1971* 


55  Bush,  J.,  M.  Chen,  and  D.  Patrick.  Cost  effectiveness  using  a  health 
status  index:  Analysis  of  the  New  York  State  phenylketonuria  screening 
program.  Unpublished  report.  University  of  California  at  San  Diego, 
Department  of  Community  Medicine,  September  197^. 


56  Cooper,  L.  G.  Metric  multidimensional  scaling  and  the  concept  of 

preference.  Los  Angeles:  Western  Management  Science  Institute,  Working 
Paper  165*  October  1970. 


67 


Bock,  R.  D.  and  L.  V.  Jones.  The  measurement  and  prediction  of  judgment 
and  choice.  San  Francisco:  Holden-Day,  1968. 
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DIRECT  METHODS  OF  WORTH  ASSESSMENT 


This  section  deals  with  methods  of  worth  assessment  in  which  the  deci¬ 
sion  maker  is  required  to  make  quantitative  judgments  rather  than  qualita¬ 
tive  judgments.  The  different  techniques  of  direct  magnitude  estimation 
can  be  categorized  into  three  classes;  first  are  those  methods  where  the 
range  of  the  scale  is  defined  by  the  researcher.  In  these  methods  the 
extreme  factor  levels  are  usually  anchored  at  the  extreme  worth  levels, 
although  only  the  scale  endpoints  need  to  be  specified.  The  range  of  the 
worth  scale  and  its  zero  point  are  arbitrarily  set  by  the  researcher,  and 
hence  the  decision  maker  specifies  his  estimates  of  worth  entirely  on  an 
interval  scale.  These  methods  will  be  called  "double  anchored  estimation 
methods . " 

A  second  class  consists  of  those  methods  where  the  researcher  provides 
only  a  single  reference  point.  The  resulting  worth  scale,  while  still  an 
interval  scale,  will  have  a  range  determined  by  the  decision  maker.  These 
methods  allow  the  decision  maker  to  give  more  information,  but  for  the  same 
reason  may  be  more  difficult  for  him  to  comprehend.  These  methods  will  be 
called  "single  anchor  estimation  methods." 

The  third  class  of  direct  estimation  methods  are  those  in  which  no 
reference  point  or  range  is  specified  by  either  the  decision  maker  or  the 
researcher.  No  research  has  been  done  on  the  use  of  these  methods  for 
worth  assessment.  There  is  some  question  as  to  whether  these  theoretical 
techniques  are  appropriate  for  worth  assessment,  and  they  will  not  be 
discussed  in  this  paper. 

Because  the  direct  estimation  methods  do  not  correspond  to  the  classical 
axioms  of  utility  (gamble  methods),  the  validity  of  their  results  has  been 
questioned.  Beach 58  investigated  the  relationship  between  the  direct 
methods  and  the  methods  which  follow  the  axioms  of  utility.  Choices  among 
bets  were  predicted  correctly  in  about  94$  of  the  cases,  using  information 
previously  found  using  the  direct  methods.  However,  about  two-thirds 
of  the  subjects  had  not  performed  in  the  gamble  methods  according  to  the 
axioms  and  had  to  be  dropped  from  the  study.  Of  the  remaining  subjects 
about  half  showed  statistically  significant  correlations  between  direct 
worth  and  worth  derived  from  the  gamble  methods.  The  study  did  not  attempt 
to  determine  which  was  the  best  method  of  worth  assessment,  so  the  incon¬ 
sistencies  could  be  a  result  of  variances  within  both  methods.  Fischer,59 
however,  found  high  correlations  between  the  direct  methods  and  the 
axiomatic  gamble  methods.  These  two  studies  indicate  that  the  direct 
methods  may  yield  results  comparable  to  the  axiomatic  methods. 


58  Beach,  B.  Direct  and  indirect  methods  for  measuring  utility.  Un¬ 
published  research  report.  University  of  Washington,  Department  of 
Psychology,  July  1972. 

59  Fischer,  G.  W.  Four  methods  for  assessing  multi-attribute  utilities: 
An  experimental  validation.  University  of  Michigan,  Engineering 
Psychology  Laboratory  Technical  Report  057230~6-T,  September  1972.  (b) 
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The  main  disadvantage  of  the  direct  methods  is  that  they  require 
numbers  from  the  decision  maker  instead  of  preference  judgments.  Most 
people  are  familiar  with  making  judgments  of  preference  or  indifference, 
but  may  be  uncomfortable  having  to  give  numbers.  It  often  requires 
training  before  they  accept  the  procedures. 

Direct  methods  have  been  used  extensively,  mainly  because  of  their 
ease  and  speed  of  application.  A  worth  function  can  be  generated  for  a 
factor  very  quickly  and  without  burdensome  calculations.  The  direct 
methods  are  integral  to  many  systems  of  worth  assessment.60'61*62 

Double  Anchored  Estimation  Method 


Double  anchored  estimation  requires  the  decision  maker  to  give  a 
number  which  directly  represents  the  worth  of  a  factor  level.  The  worths 
assessed  may  be  in  terms  of  a  reference  factor  level  and  may  involve  the 
use  of  interval  scales.  The  method: 

N  -  uses  no  probabilities 

Q  -  depends  on  quantitative  judgments 

3  -  involves  three  factor  levels  at  a  time 

E  -  works  with  either  continuous  or  discrete  factors 

N  -  provides  direct  numerical  worth 


Process  Example.  Step  1:  Rank  the  factor  levels,  or  for  continuous 
factors  specify  the  most  preferred  and  least  preferred  levels.  For  our 
example  we  will  use  efficiency  of  the  automobile  measured  in  miles  per 
gallon,  a  continuous  factor.  Assume  that  the  decision  maker  has  said  that 
30  miles  per  gallon  is  the  most  preferred  level  for  this  factor  and  that 
10  miles  per  gallon  is  the  least  preferred. 

Step  2:  Anchor  the  extreme  values  and  obtain  worths.  The  two  extreme 
factor  levels  can  be  assigned  the  extreme  values  of  the  worth  function. 

In  our  example,  worth  will  be  specified  on  a  one-to-ten  scale.  We  there¬ 
fore  assume  a  value  of  1  for  the  least  preferred  factor  level  and  10  for 


60  Churchman,  C.  W.  and  R.  L.  Ackoff.  An  approximate  measure  of  value. 
Operations  Research,  1954,  2,  172-187. 

61  Edwards,  W.  Social  utilities.  In  Proceedings  of  the  Symposium  on 
Decision  and  Risk  Ana lysis --Power ful  New  Tools  for  Management. 

Annapolis,  Md. :  U.  S.  Naval  Academy.  June  1971* 

62  Stevens,  1966,  op.  cit. 
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the  most  preferred.  The  decision  maker  is  then  presented  with  a  list  of 
the  factor  levels,  or  sample  of  factor  levels  in  the  case  of  continuous 
attributes,  similar  to  Figure  7*  He  is  then  asked  to: 


"Indicate  in  the  space  provided  a  number  between  one 
and  10  which  indicates  your  relative  preference  for 
the  factor  level  in  relationship  to  the  first  and  last 
factor  levels  which  have  been  arbitrarily  assigned 
10  and  1,  respectively. 11 

Niles  Per  Gallon  Satisfaction, 

30  JO. 

27 

24 

19 

14 

ID 


Figure  7.  Example  direct  magnitude  estimation 
work  sheet. 


The  numbers  specified  by  the  decision  maker  can  be  used  as  direct  nu¬ 
merical  worth.  They  are  direct  estimates  of  the  decisionmaker’s  true 
worth.  In  the  case  of  continuous  factors,  as  in  our  example,  graphing 
the  points  estimated  may  be  useful  for  interpolation.  Let  us  assume  that 
the  decision  maker  specified  the  numbers  9>  4  and  1.5  in  the  spaces 

in  Figure  7*  toP  to  bottom.  These  points  could  then  be  graphed  and  a 
curve  estimated  as  shown  in  Figure  8.  This  graph  may  also  be  useful  feed¬ 
back  for  the  decision  maker. 
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Figure  8.  Worth  curve  of  direct  estimates 


Variations .  A  simple  variation  is  to  list  the  factor  levels  next  to 
an  appropriate  scale  and  have  the  decision  maker  draw  a  line  from  each 
factor  to  a  point  on  the  scale  representing  his  worth  or  satisfaction. 

The  two  extreme  attribute  levels  are  anchored  to  the  extreme  worth  values: 
30  miles  per  gallon  to  10  on  the  scale,  and  10  miles  per  gallon  to  one 
(Figure  9) . 


Attribute  Level 


Satisfaction 


14  MILES 
30  MILES 
18  MILES 

ID  MILES 
26  MILES 
22  MILES 


Figure  9.  Estimation  on  a  scale 


Note  that  the  factors  have  not  been  placed  in  rank  order.  The  order  can 
be  set  arbitrarily. 

The  scale  may  be  annotated  with  descriptive  phrases  to  aid  the  deci¬ 
sion  maker.  Figure  10  shows  a  10-point  scale  annotated  with  various  levels 
of  satisfaction.  This  may  help  the  decision  maker  quantify  his  subjective 
feelings. 

Another  variation  useful  with  continuous  factors  involves  having  the 
decision  maker  draw  a  picture  of  his  worth  function.  A  graph  is  constructed 
with  the  factor  levels  on  one  axis  and  the  worth  scale  on  the  other 
(Figure  ll) .  The  extreme  worth  values,  one  and  ten  in  the  figure,  are 
anchored  to  the  extreme  factor  levels,  ten  and  30  miles  per  gallon.  The 
decision  maker  is  then  asked  to  draw  a  curve  connecting  these  two  points 
which  indicates  his  satisfaction  or  preference  with  the  intermediate 
points. 
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1 


Least 

Satisfactory 


4 


Moderately 

Satisfactory 


7 


Very 

Satisfactory 


10 


Most 

Satisfactory 


Figure  10.  Annotated  estimation  scale 


The  researcher  can  assist  the  decision  maker  with  this  task  by  coach¬ 
ing  and  by  providing  feedback.  He  can  tell  the  decision  maker  that  if  he 
feels  there  is  little  difference  in  preference  between  two  points  on  the 
miles  per  gallon  scale,  then  the  line  he  draws  between  should  be  relative¬ 
ly  flat,  and  if  there  is  a  big  difference  in  preference  the  line  should  be 
steep.  After  the  graph  is  drawn  the  researcher  can  critique  it  with  the 
decision  maker.  For  instance,  given  the  curve  in  Figure  11,  it  can  be 
noted  that  the  decision  maker  saw  little  difference  in  preference  between 
26  and  50  miles  per  gallon.  With  this  type  of  assistance  the  decision 
maker  should  be  able  to  modify  the  curve  until  it  accurately  reflects 
his  worth  function. 

For  continuous  factors,  worth  can  be  assessed  by  the  method  of  bisec¬ 
tion.  The  decision  maker  specifies  the  most  desirable  and  least  desirable 
factor  levels,  which  are  assigned  the  extreme  worth  values.  He  is  then 
asked  to  specify  a  point  which  would  yield  satisfaction  halfway  between 
the  two  points.  For  instance,  he  might  be  asked: 

"Assume  that  50  miles  per  gallon  is  worth  50  and  10  miles 
per  gallon  is  worth  0.  How  many  miles  per  gallon  would 
be  worth  25?" 

The  question  is  repeated  for  other  pairs  of  points  until  the  function 
can  be  estimated. 


Characteristics .  Cautions:  The  decision  maker  may  have  a  hard  time 
comprehending  what  is  required  of  him,  since  quantifying  subjective  feelings 
is  not  often  done.  The  researcher  must  insure  that  he  is  well  trained 
prior  to  an  actual  assessment  session.  In  addition,  using  a  terminology 
meaningful  to  the  decision  maker  will  increase  his  confidence.  For 
instance,  in  our  examples,  "satisfaction"  was  used  rather  than  ''worth" 
because  it  seemed  more  appropriate  to  the  situation.  The  researcher 
must  insure  that  the  decision  maker  understands  the  procedure.  Practice 
before  an  actual  assessment  session  may  help  the  decision  maker  understand 
what  is  required  of  him. 
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Problems  can  also  arise  if  a  factor  level  is  anchored  to  an  endpoint 
of  the  scale  before  ranking  is  done.  As  an  example,  assume  when  assessing 
the  worth  of  horsepower  that  the  researcher  anchors  the  maximum  horsepower 
to  the  high  end  of  the  scale.  The  decision  maker  might  actually  feel  that 
a  low  horsepower  is  most  satisfactory  but  could  not  indicate  this  on  the 
scale,  since  no  scale  remained  above  the  anchor  point.  In  this  case,  the 
researcher  assumed  that  more  horsepower  was  more  satisfactory  and  has 
forced  the  decision  maker  to  give  worths  that  are  unrealistic  to  him. 

It  should  be  noted  that  two  distinctly  different  techniques  have  been 
mentioned  here.  The  researcher  can  specify  a  factor  value  and  ask  the 
decision  maker  to  specify  a  worth,  either  directly  or  indirectly;  or  he 
can  specify  a  worth  and  ask  the  decision  maker  to  give  a  factor  value. 
These  two  different  techniques  may  yield  slightly  different  results. 

Equipment:  Paper  and  pencil  is  all  that  is  required.  Preprinted 

scales  may  be  useful. 

Results:  The  output  of  this  method  is  numerical  worth  that  can  be 

used  directly  in  utility  models.  In  the  case  of  continuous  factors,  the 
numbers  can  be  used  to  estimate  a  continuous  worth  function. 

Information  Needed:  The  researcher  must  know  the  nature  of  the  attri¬ 
bute  under  study,  its  measure,  and  the  range  of  the  measure.  Information 
regarding  the  total  decision  problem  is  also  helpful  in  order  to  make  the 
presentation  more  meaningful  to  the  problem  at  hand. 

Training  Required:  Since  this  method  may  be  difficult  for  the  deci¬ 
sion  maker,  he  should  be  given  a  chance  to  practice  it  before  attempting 
a  final  assessment. 


Advantages .  Direct  magnitude  estimation  is  a  quick  method  of  assessing 
worth.  The  decision  maker  is  provided  with  reference  points  to  make  his 
task  easier,  and  results  can  be  used  immediately.  The  method  is  flexible 
and  can  be  modified  to  fit  specific  situations. 


Disadvantages .  The  method  does  not  meet  the  axioms  of  classical  utility 
theory 63  since  it  requires  direct  quantification  of  subjective  feelings. 

This  may  cast  doubts  upon  the  validity  of  its  results,  although  the 
literature®*  indicates  that  the  direct  method  yields  results  comparable 
with  the  more  axiomatic  methods. 


63  Cf.  Von  Neumann  and  Morgenstern,  1944,  op.  cit. 

64  Beach,  1972,  op.  cit.;  Fischer,  1972b,  op.  cit. 
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Checking  Techniques.  The  results  can  be  checked  easily  by  performing 
the  method  twice,  at  different  times.  The  degree  to  which  the  two  sets 
of  results  agree  gives  an  indication  of  their  stability  and  accuracy.  If 
large  variances  occur  between  sessions  more  detailed  assessment  is  warranted. 

If  the  factor  under  analysis  is  continuous,  the  dispersion  of  worth 
points  from  the  estimated  curve  gives  an  indication  of  the  accuracy.  After 
the  curve  has  been  estimated,  the  decision  maker  can  be  asked  to  give  the 
worth  of  points  on  the  curve  not  already  assessed.  If  he  gives  worths 
far  from  the  expected  value  using  the  curve,  he  should  be  consulted  about 
the  discrepancy. 


Sources  of  Information.  The  theory  behind  direct  magnitude  estimation 
and  the  bisection  variation  can  be  found  in  Torgerson. 66  The  method  has 
been  described  in  Huber, 66  Eckenrode,67  and  Pardee  and  Phillips.68  Huber, 
Sahney  and  Ford  69  found  the  method  useful  in  a  study  of  hospital  ward 
evaluations.  They  used  scales  augmented  with  verbal  descriptors  (see 
Figure  10) .  Hoepfl  and  Huber 70  used  100-point  scales  with  the  method 
in  a  study  involving  evaluation  of  instructors,  and  Huber,  Daneshgar  and 
Ford  71  used  similar  scales  in  a  study  evaluating  prospective  jobs.  Sellin 
and  Wolfgang  72  used  this  method  in  evaluating  the  judged  seriousness  of 
crimes . 


65  Torgerson,  1958;  op.  cit. 

66  Huber,  1974;  op.  cit. 

67  Eckenrode,  R.  Weighting  multiple  criteria.  Management  Science,  1965; 

12,  180-192. 

68  Pardee,  F.  S.  and  C.  T.  Phillips.  The  utility  of  utility  theory  in 
regional  transportation  mix  analysis.  Presented  at  the  36th  National 
Meeting  of  the  Operations  Research  Society  of  America,  Miami,  August 
1970.  (AD  711-497) 

69  Huber,  G.,  V.  Sahney  and  D.  Ford.  A  study  of  subjective  evaluation 
models.  Behavioral  Science,  1969;  14,  483-489. 

70  Hoepfl,  R.  and  G.  Huber.  A  study  of  self-explicated  utility  models. 
Behavioral  Science,  1970;  15 t  408-414. 

71  Huber,  G.,  R.  Daneshgar  and  D.  Ford.  An  empirical  comparison  of  five 
utility  models  for  predicting  job  preferences.  Organizational  Behavior 
and  Human  Performance,  1971;  §j  267“282. 

72  Sellin,  T.  and  M.  E.  Wolfgang.  The  measurement  of  delinquency.  New  York: 
Wiley,  1964. 
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The  graphical  variation  was  tested  by  Pai,  Gustafson  and  Kiner73  and 
was  used  by  Gustafson  et  al.74  to  develop  a  burn  severity  index.  Five 
factors  were  developed  which  appeared  important  in  evaluating  burn  severity: 
size  of  full  thickness  burn,  size  of  partial  thickness  burn,  age,  past 
medical  history,  and  location  of  burn.  A  severity  function  was  developed 
for  each  of  these  factors,  the  factors  were  weighted,  and  their  weighted 
sum  was  used  as  a  burn  severity  index.  The  first  three  factors  can  be 
described  as  continuous  and  were  analyzed  by  the  graphical  method.  Figure 
12  is  an  example  of  a  graph  drawn  by  a  physician  to  show  the  "satisfaction 
function"  of  age  on  severity  of  burn.  The  graphs  drawn  by  several  physicians 
were  combined  to  get  a  severity  function  of  each  of  these  factors.  The 
other  two  factors  and  the  factor  weights  were  developed  using  a  single 
anchored  estimation  technique.  The  model  developed  correlated  well  with 
both  physician  estimates  and  survival  rate. 

Davidson  75  describes  the  technique  as  used  in  land-use  decision  making. 
Levine  et  al.  78  and  McKendry  et  al.77  both  used  direct  estimation  methods 
in  their  studies  of  surveillance  information  error  costs.  Miller78  has 
suggested  the  use  of  the  graphical  variation  in  R&D  project  evaluation. 

Direct  magnitude  estimation  is  used  as  one  part  of  the  Churchman-Ackof f 
method  of  worth  assessment79  and  was  used  by  Dalkey,  Brown  and  Cochran so 
in  explicating  subjects'  self-ratings  in  a  Delphi  study. 


73  Pai,  G.,  D.  H.  Gustafson  and  G.  Kiner.  Comparison  of  three  non-risk 
methods  for  determining  a  preference  function  for  money.  Unpublished 
report,  University  of  Wisconsin,  January  1971* 

74  Gustafson,  D.  H.,  I.  Feller,  K.  Crane,  and  D.  Holloway.  A  decision 
theory  approach  to  measuring  severity  in  illness.  Unpublished  report, 
University  of  Wisconsin,  1971* 

75  Davidson,  F.  Utility  and  decision  theory  in  a  planning  environment. 
Unpublished  report,  University  of  Virginia,  Mclntire  School  of  Commerce, 

1970. 

76  Levine,  J.  M.,  J.  B,  Feallock,  R.  Sadacca  and  R.  Andrews.  Method  for 
quantifying  subjective  costs  of  large  numbers  of  image  interpretation 
errors.  ARI  Technical  Research  Note  2l8,  November  1969-  (AD  704  706) 

77  McKendry,  J.  M. ,  P.  C.  Harrison,  A.  H.  Birnbaum  and  R.  Sadacca.  Estimat¬ 
ing  the  value  of  surveillance  information  using  error  cost  matrices. 

ARI  Technical  Research  Note  184.  June  1967.  (AD  667  390) 

78  Miller,  1967*  op.  cit. ;  1969,  op.  cit. 

79  Churchman  and  Ackoff,  1954,  op.  cit. 

80  Dalkey,  N.,  B.  Brown  and  S.  Cochran.  The  Delphi  method:  Use  of  self 
ratings  to  improve  group  estimates.  Rand  Corp.  Memo  RM-6115~PR, 

November  I969.  (AD  698  755) 
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Single  Anchored  Estimation  Method 

The  basic  method  described  here  is  based  on  ratio  comparisons  of  two 
factors.  Each  factor  level  is  compared  to  a  standard,  and  the  decision 
maker  is  asked  to  give  a  number  which  represents  the  ratio  of  the  worths. 
The  method: 

N  -  uses  no  probabilities 
Q  -  depends  upon  quantitative  judgments 
2  -  involves  two  factor  levels  at  a  time 

E  -  works  with  either  continuous  or  discrete  factors 
N  -  provides  direct  numerical  worth 


Process  Example.  Step  1:  First,  the  decision  maker  must  rank  order 
the  factor  levels  under  consideration.  For  continuous  factors,  he  should 
specify  the  least  and  most  desirable  levels  the  factors  can  take.  As 
an  example,  assume  that  the  relative  worths  of  five  models  are  needed  in 
a  decision  to  purchase  an  automobile  (Figure  4).  Assume  that  the  decision 
maker  has  said  that  his  preference  is  in  the  order  of  convertible,  two-door 
coupe,  two-door  hardtop,  four-door  hardtop  and  station  wagon,  with  convert¬ 
ible  being  the  most  preferable. 

Step  2:  Compare  the  worths  of  all  factor  levels  with  a  randomly  chosen 
level  one  by  one.  One  factor  level  is  chosen  to  act  as  the  reference. 

The  other  factor  levels  are  compared  to  this  factor  level  in  an  arbitrary 
order.  Step  1  indicated  which  of  the  pair  is  preferred;  in  this  step 
the  decision  maker  is  asked  to  specify  how  much  more  it  is  preferred  in 
terms  of  a  ratio.  In  the  example,  assuming  the  two-door  hardtop  is  the 
reference  factor,  the  decision  maker  would  be  asked: 

"How  much  more  is  a  convertible  worth  to  you  compared 
to  a  two-door  hardtop?"  (i.e.,  twice?  three  times? 
one  and  one-half  times?) 

He  might  answer  that  a  convertible  is  worth  one  and  one-half  times  as 
much  as  a  two-door  hardtop  to  him.  This  procedure  is  repeated  for  all 
the  other  factors. 

Step  5:  Assign  the  reference  factor  level  an  arbitrary  worth.  In 
our  example,  a  two-door  hardtop  will  be  assigned  a  worth  of  50  by  the 
researcher.  Then  numerical  points  can  be  assigned  for  each  of  the  other 
factor  levels  according  to  the  ratio  of  the  worth  of  those  factor  levels 
to  the  reference  factor  level.  In  our  example,  convertible  would  be 
assigned  a  worth  of  1.5x50=  75* 

Once  the  points  for  all  the  factors  have  been  assigned,  they  could  be 
represented  on  a  scale  to  provide  the  decision  maker  with  visual  feedback. 
Figure  13  shows  a  possible  scale  where  convertible  is  clearly  most  pre¬ 
ferred,  station  wagon  is  clearly  least  preferred  and  the  other  three 
levels  are  grouped  in  the  middle. 
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Figure  12.  Worth  function  drawn  by  a  physician  for  developing  a  severity  of  burn  index. 

(From  Gustafson,  D.  H.,  I.  Feller,  K.  Crane,  and  D.  Holloway.  A  decision  theory  approach  to  measuring  severity 
in  illness.  Unpublished  report.  University  of  Wisconsin,  1971.) 


Two-door  Coupe 
Four-door  Hardtop 


Figure  13.  Example  worth  scale 


Variations .  One  variation  involves  estimating  factor  level  magnitudes 
rather  than  worth  ratios.  This  can  be  used  only  with  continuous  factors. 
Using  miles  per  gallon  as  an  example:  first,  10  miles  per  gallon  would 
be  arbitrarily  assigned  a  worth--assume  a  value  of  10.  The  decision  maker 
is  then  asked  to  specify  how  many  miles  per  gallon  would  be  twice  as 
satisfactory  as  10  miles  per  gallon.  This  factor  level  is  then  assigned 
a  value  of  20.  The  process  is  then  repeated  asking  for  a  value  three 
times  as  satisfactory  as  10  miles  per  gallon,  or  twice  as  satisfactory  as  the  Laat 
specified  value.  These  various  points  can  then  be  plotted  and  a  curve  estimated. 

A  similar  variation  involving  a  gamble  method  is  described  in  the  next  section. 

Most  of  the  variations  described  in  the  double  anchored  estimation 
section  can  be  modified  and  used  with  the  single  anchored  estimation 
method.  However,  care  must  be  taken  to  insure  that  the  decision  maker  is 
not  constrained  by  the  given  scale.  For  instance,  assume  that  a  reference 
level  is  anchored  to  nine  on  a  ten-point  scale.  If  the  decision  maker 
feels  that  some  other  level  is  three  times  as  satisfactory  as  the  reference, 
he  will  be  unable  to  show  this  on  the  scale.  The  use  of  log  scales  may 
help  alleviate  this  problem, 81  since  a  wide  range  of  values  are  possible 
while  small  differences  can  still  be  noted  (Figure  14). 


S1  Gustafson  et  al.,  1971*  °P*  cit 


ATTRIBUTE  LEVEL 


Figure  14.  Single  anchored  estimation  on  a  log  scale 
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Characteristics .  Cautions:  This  method  is  only  useful  when  the  deci¬ 
sion  maker  is  familiar  with  the  estimation  of  ratios.  For  example,  when 
asked  to  compare  a  two-door  hardtop  with  a  convertible,  he  might  estimate 
the  worth  as  80$  of  that  of  a  convertible.  But  when  asked  the  question 
the  other  way  around,  he  might  estimate  that  a  convertible  is  worth  1.5 
times  as  much  as  a  two-door  hardtop.  Note  that  if  a  convertible  is  assumed 
to  have  a  worth  of  100,  the  worth  of  the  two-door  hardtop  is  80  in  the 
first  case  and  67  in  the  second.  Training  in  the  use  of  ratios  may  be 
needed  to  prevent  such  errors  as  an  order  bias. 

Results:  The  output  of  this  method  is  a  direct  numerical  worth  of 
factor  levels  on  an  arbitrary  scale.  The  endpoints  will  vary  according  to 
the  one  point  fixed  by  the  researcher. 

Information  Needed:  The  only  information  needed  is  the  factor  levels 
whose  worths  are  to  be  measured  (if  discrete)  and  the  range  of  values  in 
the  case  of  continuous  factors. 

Equipment:  No  special  equipment  is  needed  for  this  procedure.  Paper 
and  pencil  may  be  required  in  some  of  the  variations. 


Advantages .  Ease  of  Application:  Once  accepted,  this  is  probably  one 
of  the  easiest  methods  of  estimating  the  worth  of  factor  levels  where 
probability  is  not  involved.  It  demands  very  little  of  the  researcher  or 
decision  maker. 

No  Fixed  Scales:  Since  the  decision  maker  is  not  forced  to  evaluate 
the  relative  worths  of  different  factor  levels  on  a  fixed  scale  (for 
example,  on  a  10-100  scale),  there  is  no  f,endpoint  problem, 11  i.e.,  the 
decision  maker  would  not  have  to  feel  uneasy  about  choosing  values  near 
10  or  100. 


Disadvantages .  Acceptability:  This  method  is  quite  unpopular  with 
decision  makers.  Many  feel  more  uncomfortable  estimating  in  ratios  than 
in  any  other  mode.  This  can  be  alleviated  by  training  and  discussion  but 
this  is  a  slow,  time-consuming  process. 

Accuracy:  The  decision  maker  estimating  the  ratios  may  not  be  able 
to  judge  ratios  well.  Some  people  tend  to  overestimate  ratios  while  others 
tend  to  underestimate  them.  Results  of  a  study  in  the  area  of  subjective 
probability  estimation  82  where  actuarial  data  exist  to  check  assessment 
validity  indicate  that  this  type  of  range  error  may  be  no  greater  and 
possibly  less  than  with  other  direct  estimation  techniques. 


82  Stauss,  F.  Comparison  of  five  estimation  techniques  for  subjective 
probability.  Unpublished  Master's  Thesis,  University  of  Wisconsin, 
Department  of  Industrial  Engineering,  1972. 
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Sensitivity:  This  method  is  not  very  sensitive  to  small  differences 
in  ratios.  For  example,  the  decision  maker  may  not  be  able  to  differentiate 
between  14/16  and  I5/I6  (about  a  six-point  difference  on  a  100-point  scale). 
The  result  is  a  certain  lack  of  resolution  or  degree  of  error  inherent  in 
the  method. 


Checking  Techniques.  One  way  to  check  on  the  decision  maker's  estimation 
is  to  have  him  evaluate  all  ( or  a  subset  of)  the  factor  levels  again,  this 
time  relative  to  some  other  factor  level  other  than  the  original  reference 
point.  In  our  example,  the  decision  maker  might  be  asked  to  estimate  the 
relative  worth  of  a  station  wagon  compared  to  a  four-door  hardtop.  These 
new  estimates  should  be  nearly  the  same  ratio  as  the  ratio  of  the  worths 
developed  orginally.  If  the  researcher  finds  gross  inconsistencies,  he 
should  point  them  out  to  the  decision  maker  and  ask  him  to  reconsider. 

This  procedure  could  be  repeated  as  many  times  as  desired  using  different 
factors  as  bases. 


Source  of  Information.  The  theoretical  background  for  this  method  is 
found  in  Ekman,  s3  Coombs  and  Komorita  84  and  Coombs.  S5  Gustafson  et  al.  S6 
used  the  method  in  developing  the  severity-of-burn  index  mentioned  earlier, 
in  which  two  of  the  five  factors  assessed  were  evaluated  using  a  single  anchored 
estimation  technique--location  of  burn  and  past  medical  hostory.  To  evaluate 
the  effects  of  location,  various  locations  were  written  on  cards  and  the 
decision  makers  sorted  these  in  order  of  importance.  They  then  indicated 
how  much  more  satisfied  they  would  be  with  a  patient  burned  in  the  first 
location  than  with  a  patient  burned  in  each  of  the  other  locations.  They 
gave  responses  on  a  log  scale.  The  worst  location  was  then  arbitrarily 
assigned  a  value  of  100  and  every  other  location  was  valued  in  relation 
to  this.  In  a  similar  manner,  past  medical  history  and  the  criteria 
weights  were  assessed.  The  model,  which  included  the  three  factors  evalu¬ 
ated  using  a  double  anchored  estimation  technique,  correlated  highly  with 
both  survival  rate  and  physician  estimates. 


S3  Ekman,  G.  Two  generalized  ratio  scaling  methods.  Journal  of  Psychology, 

1958,  45,  287-295. 

84  Coombs,  C.  H.  and  S.  S.  Komorita.  Measuring  utility  of  money  through 
decisions.  American  Journal  of  Psychology,  1958*  71.  383*389* 

s5  Coombs,  C.  H.  Inconsistency  of  preferences:  A  test  of  unfolding  theory. 
In  W.  Edwards  and  A.  Tversky  (Ed.),  Decision  Making.  Baltimore: 

Penguin,  1967* 

S6  Gustafson  et  al.,  1971*  op.  cit. 
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Klahr  87  used  a  variant  of  this  method  for  use  with  a  model  to  pre¬ 
dict  college  admission  decisions.  Pai,  Gustafson  and  Kiner  88  found  the 
results  of  this  method  correlated  highly  with  the  results  of  other  methods. 
Huber  and  Gustafson89  used  the  method  on  several  attributes  for  developing 
severity  of  illness  indices.  In  another  example,  Beach 90  used  the  method 
in  a  comparison  of  direct  and  indirect  worth  assessment  techniques. 
Gustafson,  Kramer  and  Pai91  used  the  method  in  a  study  of  research  and 
development  project  selection  strategies,  and  Stimson92  employed  the 
technique  in  his  studies  to  improve  the  decision  making  process  for  funding 
public  health  grants.  Finally,  Galanter93  described  a  variation  of  the 
method. 


GAMBLE  METHODS  OF  WORTH  ASSESSMENT 

The  methods  using  gambles  as  a  basis  generally  follow  the  axioms  of 
utility  theory94  and  therefore  have  been  quite  popular.  These  axioms 
detail  a  set  of  rules  which,  if  followed  in  choice  situations,  will 
guarantee  the  existence  of  a  worth  function  which  describes  the  decision 
maker. 

A  gamble  or  wager  is  defined  as  a  situation  where  each  one  of  a  set 
of  outcomes  can  occur  with  a  given  probability.  For  instance,  winning 
$10  or  losing  $5  based  on  the  flip  of  a  coin  is  a  wager  where  the  two 
outcomes,  "win  $10"  and  "lose  $5"  occur  with  probabilities  of  1/2.  The 
methods  rely  on  the  decision  maker's  ability  to  choose  between  such  a 
wager  and  an  alternative  with  no  risk  involved,  a  "sure  thing."  Extending 
the  above  wager,  the  decision  maker  would  choose  between  (a)  a  wager 
where  the  probability  of  winning  $10  is  l/2  and  the  probability  of  losing 
$5  is  l/2;  or  (b)  winning  $2  for  sure. 


87  Klahr,  D.  Decision  making  in  a  complex  environment:  The  use  of  sim¬ 
ilarity  judgments  to  predict  preferences.  Management  Sc ience ,  1969, 

15,  595-618. 

88  Pai,  Gustafson,  and  Kiner,  1971,  op.  cit. 

89  Huber,  G.  and  D.  H.  Gustafson.  Some  efforts  to  apply  behavioral  deci¬ 
sion  theory  in  the  medical  care  field.  Unpublished  report,  University 
of  Wisconsin,  September  1971 • 

90  Beach,  1972,  op.  cit. 

91  Gustafson,  D.  H.,  G.  Kramer,  and  G.  Pai.  A  weighted  aggregate  approach 

to  R&D  project  selection.  AIIE  Transactions,  I97I,  22-31. 

92  Stimson,  D.  H.  Utility  measurement  in  public  health  decision  making. 
Management  Science,  I969,  16*  17“30. 

93  Galanter,  F.  The  direct  measurement  of  utility  and  subjective  probability. 
American  Journal  of  Psychology,  1962,  75,  208-220. 

94  Von  Neumann  and  Morgenstern,  1944,  op.  cit. 
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There  are  two  different  ways  for  using  these  choices  to  develop  worth 
functions.  First,  the  probabilities  in  the  wager  can  be  varied  until  the 
decision  maker  is  indifferent  between  wager  and  sure -thing.  At  this 
point,  it  is  assumed  that  the  expected  worth  of  the  wager  and  worth  of  the 
sure-thing  are  equal.  These  methods  will  be  called  the  variable  probability 
methods.  Second,  the  probabilities  in  the  wager  can  be  held  constant  and 
one  of  the  outcomes  can  be  varied  until  the  decision  maker  is  indifferent. 
These  will  be  called  the  constant  probability  methods. 

The  fact  that  the  methods  are  generally  axiomatically  valid  has  been 
their  main  advantage.  In  addition,  they  incorporate  the  notion  of  risk  in 
the  analysis,  which  many  believe  makes  the  method  more  realistic,  since 
most  real  world  situations  involve  a  certain  amount  of  uncertainty. 

The  main  disadvantage  of  these  methods  is  that  they  are  difficult  and 
time-consuming  to  administer.  Many  different  wagers  must  be  constructed 
and  analyzed,  and  the  decision  maker  is  forced  to  evaluate  rather  complex 
situations  and  to  evaluate  them  in  a  specific  manner.  That  is,  he  must 
follow  the  axioms  and  try  to  maximize  his  expected  worth.96 

The  methods  involve  presenting  many  hypothetical  situations  which'  may 
appear  to  be  very  game-like  to  the  decision  maker.  This  may  jeopardize 
his  serious  participation  in  the  methods,  and  training  and  support  may 
be  necessary  to  overcome  the  problem. 


Variable  Probability  Method 

This  method  involves  the  comparison  of  wagers  by  the  decision  maker 
to  generate  his  worth  function.  Probabilities  in  the  wagers  are  varied 
until  the  decision  maker  is  indifferent  between  the  wager  and  a  sure- 
thing  option.  These  indifference  points  define  a  worth  function  on  an 
arbitrary  interval  scale.  The  method: 

P  -  used  probabilities 

I  -  depends  upon  indifference  judgments 
5  -  involves  three  factor  levels  at  a  time 

E  -  works  with  either  continuous  or  discrete  factors 
N  -  provides  direct  numerical  worth 


Process  Example.  Step  1:  The  decision  maker  must  rank  order  the  levels 
of  discrete  factors,  or  at  least  specify  a  most  preferred  and  least  preferred 
factor  level.  For  continuous  factors,  only  the  most  and  least  preferred 
factor  levels  are  necessary.  An  an  example,  assume  that  the  worths  of 


96  See  Edwards,  W.  Utility,  subjective  probability,  their  interaction  and 
variance  preferences.  Journal  of  Conflict  Resolution,  1962,  6,  42-51. 
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various  models  are  needed  in  a  decision  to  purchase  an  automobile.  There 
are  five  possible  models:  station  wagon,  convertible,  two-door  coupe, 
two-door  hardtop  and  four-door  hardtop.  Assume  that  the  decision  maker 
said  he  most  prefers  a  convertible  and  least  prefers  a  station  wagon. 

Step  2:  Choose  a  scale  and  assign  numerical  values  to  the  most 
preferred  and  least  preferred  factor  levels.  The  researcher  should 
determine  the  required  range  of  the  worth  scale.  This  is  arbitrary  and 
chosen  for  convenience,  since  points  on  the  scale  have  no  physical  meaning; 
only  the  distances  between  points  are  useful.  Assume  that  in  our  example 
a  scale  from  +10  to  -10  was  chosen,  with  -10  being  least  preferred  and 
+10  being  most  preferred.  Thus,  a  convertible  will  be  assigned  a  worth 
of  +10  and  a  station  wagon  -10.  It  should  be  noted  that  any  range  of 
values  is  justifiable.  For  instance,  0-100  is  often  used. 

Step  3:  Construct  a  wager  and  present  a  choice  situation  to  the  deci¬ 
sion  maker.  A  wager  is  constructed  which  involves  the  most  preferred  and 
least  preferred  factor  levels.  The  wager  might  be  getting  a  convertible 
with  a  probability  of  .1  or  getting  a  station  wagon  with  a  probability  of 
•9*  The  decision  maker  is  then  presented  with  a  choice,  either  the  wager 
or  one  of  the  other  factor  levels  as  a  sure-thing,  and  asked  which  he 
prefers.  In  our  example  he  would  be  asked: 

"Would  you  prefer  to  accept  a  wager  where  the  probabil¬ 
ity  of  getting  a  convertible  is  .1  and  the  probability 
of  getting  a  station  wagon  is  .9*  or  would  you  prefer 
to  have  a  two-door  coupe  for  sure?" 

The  objective  of  this  step  is  to  find  a  point  at  which  the  decision 
maker  is  indifferent  between  the  wager  and  the  sure-thing.  If  he  clearly 
prefers  one  or  the  other,  the  probabilities  must  be  varied  until  he  is 
indifferent,  i.e.,  feels  that  the  wager  and  sure-thing  are  equally 
acceptable.  Changing  the  probabilities  will  change  the  expected  worth 
of  the  wager.  If  the  probability  of  getting  the  most  preferred  factor 
level  is  increased  (with  a  similar  decrease  in  probability  of  the  least 
preferred  factor  level)  the  expected  worth  of  the  wager  will  increase. 
Therefore,  if  the  sure-thing  is  preferred,  it  is  worth  more  than  the  wager. 
The  probability  of  the  most  preferred  factor  level  (a  convertible  in  this 
example)  must  be  increased  to  increase  the  expected  worth  of  the  wager. 

If  the  wager  is  preferred,  its  expected  worth  must  be  reduced  by  decreasing 
the  probability  of  the  most  preferred  factor  level  and  increasing  the 
probability  of  the  least  preferred  factor  level.  In  this  manner,  the 
expected  worth  of  the  wager  is  modified  until  the  decision  maker  believes 
it  to  be  equal  to  the  worth  of  the  sure-thing  option. 

At  the  indifference  point,  the  decision  maker  feels  that  the  expected 
worth  of  the  wager  and  the  worth  of  the  sure-thing  are  equal,  or  at  least 
so  close  as  to  be  indistinguishable.  In  other  words,  the  chances  of  get¬ 
ting  the  most  preferred  factor  level  (convertible)  are  just  high  enough 
and  the  chances  of  getting  the  least  preferred  factor  level  ( station  wagon) 
are  just  low  enough  so  that  it  is  almost  ,lworth  it"  to  take  the  gamble 
rather  than  settle  for  an  intermediate  factor  level  (two-door  coupe). 
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Let  us  assume  that  the  decision  maker  is  indifferent  when  the  probabil¬ 
ity  of  a  convertible  is  .95  and  the  probability  of  a  station  wagon  is  .05. 
The  worth  of  a  two-door  coupe  is  then  equal  to  .95  times  the  worth  of  a 
convertible  plus  .05  times  the  worth  of  a  station  wagon.  In  symbols  this 
is : 

W  (two-door  coupe)  =  .95  W  (convertible)  (l) 

+.05  W  (station  wagon) 

Since  we  assigned  values  to  the  worths  of  a  convertible  and  station  wagon 
in  Step  2,  we  can  calculate  the  relative  value  of  the  two-door  coupe. 

W  (two-door  coupe)  =  .95  (10)  +  .05  (-10)  =  9.0  (2) 

Step  4:  Vary  the  sure-thing.  A  worth  is  assigned  to  the  sure-thing 
in  Step  so  that  step  is  repeated  using  a  different  sure-thing  outcome. 
This  is  continued  until  all  factor  levels  other  than  most  and  least  pre¬ 
ferred  have  been  the  sure-thing  option  and  assigned  a  worth.  In  the  case 
of  continuous  factors,  the  process  is  continued  until  enough  values  have 
been  generated  to  estimate  the  relationship  between  factor  levels  and 
worth,  one  point  being  determined  for  each  replication  of  Step  3*  In  our 
example,  Step  3  must  be  repeated  twice,  once  using  two-door  hardtop  and 
once  using  four-door  hardtop  as  the  sure-thing  options.  Assume  that  the 
decision  maker  felt  that  the  two-door  hardtop  was  indifferent  to  the  wager 
when  the  probability  of  getting  a  convertible  was  .7  and  a  four-door  hard¬ 
top  was  indifferent  to  the  wager  when  the  probability  was  .25.  We  can 
calculate  from  this  the  worth  of  these  models,  or  +4  and  -5,  respectively: 

W  (two-door  hardtop)  =  .7(10)  +  .3(~10)  *  4.0  (3) 

W  (four-door  hardtop) =  .25(10)+  «75( "10 ) *  "5*0 

It  should  be  noted  that  using  zero  as  one  end  of  the  scale  reduces  the 
number  of  calculations,  since  either  the  worth  of  the  most  preferred  or 
worth  of  the  least  preferred  factor  level  will  be  zero  and  can  be  left 
out  of  the  calculations*  For  continuous  factors,  the  points  can  be  plotted 
and  a  curve  estimated. 


Variations.  In  constructing  choices  for  the  decision  maker  it  is 
possible  to  use  other  than  the  most  and  least  preferred  factor  levels  in 
the  wagers.  The  only  requirement  is  that  the  sure-thing  be  more  preferred 
than  one  of  the  wager  factor  levels  and  less  preferred  than  the  other. 

It  can  be  easily  seen  that  if  the  sure-thing  is  preferred  to  both  factor 
levels  in  the  wager,  it  will  always  be  preferred  to  the  wager  regardless 
of  the  probabilities.  The  opposite  holds  true  if  both  wager  factor  levels 
are  preferred  to  the  sure-thing.  No  point  of  indifference  can  ever  be  found 
in  either  case.  One  common  method  is  to  use  three  adjacent  factor  levels 
in  rank  order.  Using  four-door  hardtop  and  two-door  coupe  in  the  wager 
against  two-door  hardtop  as  a  sure-thing  would  be  such  a  situation.  The 
solution  of  the  resulting  worth  relationships  (Equation  l)  will  not  contain 
common  terms  and  sol*  ing  them  will  be  slightly  more  difficult. 
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Characteristics .  Cautions:  This  method  is  applicable  only  with  sub¬ 
jects  who  are  familiar  with  or  capable  of  understanding  probability.  If 
this  condition  is  violated,  the  resulting  worth  values  will  be  questionable. 
For  example,  assume  that  a  decision  maker  values  two-door  hardtop,  10; 
two-door  coupe,  30;  and  convertible,  60;  and  we  present  him  with  the 
following  choice: 

'tyould  you  prefer  a  wager  where  the  probability  of 
getting  a  convertible  is  .7  and  the  probability  of 
getting  a  two-door  hardtop  is  .3,  or  would  you  prefer 
a  two-door  coupe  for  sure?11 

He  may  uhderestimate  the  meaning  of  .7  and  overestimate  what  is  meant  by 
•3  and  be  indifferent  between  the  choice.  This  would  lead  us  to  believe 
that  he  values  a  two-door  coupe  at  45,  though  he  really  values  it  at  3^. 
Training  the  decision  maker  in  the  use  of  probability  usually  reduces 
this  problem. 

Another  aspect  of  this  method  that  may  cause  problems  is  that  the  prob¬ 
abilities  in  the  wager  must  add  to  one.  In  other  words,  the  wager  must 
be  exhaustive.  For  instance,  if  the  two  probabilities  added  to  .7,  there 
would  be  a  probability  of  .3  that  something  other  than  the  two  outcomes 
would  occur.  Since  the  ’third*  outcome  is  undefined,  errors  will  occur. 

The  "indifference  point"  is  sometimes  an  interval;  the  decision  maker 
may  be  indifferent  over  a  relatively  wide  range  of  probabilities.  In  these 
cases  it  is  worthwhile  to  continue  varying  the  probabilities  even  though 
an  indifferent  situation  has  been  found.  In  this  way  an  approximate  range 
of  indifference  can  be  found  and  its  central  point  used  in  the  worth 
calculations . 96 

Finally,  the  decision  maker  may  like  to  gamble  or  be  afraid  of  gambling. 
His  decision  would  be  influenced  by  this  attitude  toward  risk. 

Equipment:  A  device  for  graphically  portraying  the  probabilities  to 
the  decision  maker  is  recommended,  since  there  are  problems  with  using 
stated  probabilities.  Figure  15  shows  two  different  types:  the  probability 
wheel  (a)  and  the  probability  bar  (b).  The  wheel  is  constructed  using 
two  colored  circles  that  are  interwoven  and  pinned  at  the  center  so  they 
can  be  rotated  relative  to  each  other  to  vary  the  size  of  the  darkened 
area.  In  Figure  15,  the  wheel  shows  graphically  the  wager  probabilities 
of  .30  and  .70.  The  probability  bar  works  like  a  slide  rule,  except 
that  the  relative  sizes  of  the  two  colors  represent  probabilities  for  the 
decision  maker. 


96  Beach,  1972,  op.  cit. 
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Figure  15.  Devices  for  portraying  probability 


Results:  The  output  of  this  method  is  numbers  whose  values  represent 

the  relative  worth  of  the  factor  levels.  The  range  and  origin  of  the 
numbers  are  arbitrarily  set  by  the  researcher.  If  the  factor  is  contin¬ 
uous,  the  method  will  generate  points  from  which  the  worth  curve  can  be 
estimated . 

Information  Needed:  The  range  and  precise  description  of  the  factor 
being  studied  is  the  main  information  needed.  It  is  recommended  that  the 
decision  maker1 s  ability  with  probability  be  at  least  estimated  before 
attempting  this  method.  This  can  be  done  most  directly  by  asking  the 
decision  maker  for  his  own  self-evaluation,  or  alternatively  by  having 
him  judge  known  probabilities. 

Training  Required:  Allowing  the  decision  maker  to  practice  the  use 
of  this  method  before  an  actual  assessment  may  be  necessary  to  insure  he 
understands  the  technique.  Training  him  in  the  meaning  of  probability 
may  be  beneficial.  Hoffman  and  Peterson97  describe  a  method  for  training 
people  to  be  good  probability  assessors. 


Advantages .  Axiomatic:  The  method  is  theoretically  appealing  since 
it  is  derived  from  the  axioms  of  utility.98 


Disadvantages .  Probability  Interpretation:  The  main  problem  with 
the  method  is  that  different  individuals  may  interpret  or  react  differently 
to  the  same  probability.  People  appear  to  make  choices  in  wager  conditions 
on  the  basis  of  perceived  probabilities,  rather  than  the  stated  objective 
probabilities.99'100  Since  worths  are  generated  using  objective  proba¬ 
bilities,  bias  is  introduced  into  the  method  according  to  how  the  decision 
maker  perceived  the  probabilities.  Fishburn101  has  suggested  that  a  sub¬ 
jective  function  of  objective  probability  should  be  determined  for  the 
decision  maker,  and  that  this  be  used  in  the  method  when  calculating 
worths . 


07 Hoffman,  J.  and  C.  Peterson.  A  scoring  rule  to  train  probability 
assessors.  University  of  Michigan,  Engineering  Laboratory  Technical 
Report  037230-4-T,  September  1972. 

98 Von  Neumann  and  Morgens tern,  1944,  op.  cit. 

"Edwards,  W.  Probability-preference  among  bets  with  differing  expected 
values.  American  Journal  of  Psychology,  1954,  6j[,  56-57* 

100Edwards,  1962,  op.  cit.;  Slovic  and  Lichtenstein,  1968,  op.  cit. 

101  Fishburn,  1964,  op.  cit. 
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Inveterate  Gamblers:  Some  people  enjoy  gambling  and  will  tend  to 
choose  a  gamble  over  a  sure-thing,  even  when  the  sure-thing  is  obviously 
worth  more.  The  thrill  of  the  gamble  tends  to  increase  the  worth  of  the 
factors  within.  If  the  decision  maker  being  studied  is  this  type,  this 
method  will  be  strictly  inappropriate.  In  the  opposite  extreme,  a  decision 
maker  may  be  adverse  to  risk  and  always  choose  the  sure-thing,  again 
invalidating  the  method. 

Acceptability:  Decision  makers  may  rebel  at  the  idea  of  playing  a 
'game.1  If  forced  to  continue  they  may  not  concentrate  on  the  task.  The 
method  should  be  used  primarily  in  situations  where  the  hypothetical 
situation  posed  is  realistic. 


Checking  Techniques.  There  are  two  basic  techniques  for  checking  the 
worths  generated  by  this  method.  The  first  involves  presenting  the  decision 
maker  with  redundant  wagers  after  worth  has  been  assessed.  For  instance, 
assuming  we  have  completed  the  assessment  of  model  worth,  we  might  ask 
the  decision  maker: 

"Would  you  prefer  to  accept  a  chance  wager  where  the 
probability  of  getting  a  two-door  coupe  is  .8  and  the 
probability  of  getting  a  four-door  hardtop  is  .2, 
or  would  you  prefer  a  two-door  hardtop  for  sure?" 

He  should  prefer  the  wager  since  we  assume  the  expected  worth  of  the  wager 
is  .2  times  the  worth  of  a  four-door  hardtop  plus  .8  times  the  worth  of 
a  two-door  coupe,  or  6.2  (Equation  l)  and  the  worth  of  a  two-door  hardtop 
is  4.  If  the  decision  maker  does  not  prefer  the  wager,  the  discrepancy 
must  be  cleared  up  before  using  the  worths. 

The  second  method  of  checking  is  to  measure  the  "width"  of  the  indif¬ 
ference  point.  Specifically,  there  will  usually  be  a  range  of  probabili¬ 
ties  in  the  wager  within  which  the  decision  maker  is  indifferent  because 
of  his  inability  to  perceive  precisely  his  own  values  and  the  probabilities 
presented.  A  wide  range  for  the  indifference  point  may  indicate  that  the 
decision  maker  is  not  sure  about  the  worths  and  may  change  as  his  values 
strengthen.  The  decision  maker  may  have  to  be  questioned  further  if  the 
points  are  wide,  so  that  he  may  increase  his  confidence  in  his  own  values. 


Source  of  Information.  The  general  theory  for  this  method  was  developed 
by  Von  Neumann  and  Morgans  tern 102  and  has  been  described  by  Mosteller  and 


102 Von  Neumann  and  Morganstern,  1944,  op.  cit. 
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Nogee,  103  Raiffa, 104  and  Swalm.  105  The  Decision  Analysis  Group  of  the 
Stanford  Research  Institute  (especially  Carl  Spetzler)  makes  extensive 
use  of  this  method  using  the  probability  wheel  (Figure  15) • 

Marquardt,  Makens  and  Larzelere  106  used  the  method  to  determine  the 
worth  of  brand  names  on  turkeys.  They  compared  a  common  brand-name  turkey 
with  an  unknown  turkey.  The  subjects  were  told  that  both  turkeys  were 
worth  $4.00.  The  subjects  were  asked  to  choose  between  two  situations. 

In  the  first  they  would  receive  the  brand  name  turkey  eight  times  out  of 
ten  or  two  dollars  two  times  out  of  ten.  (This  was  the  ’Vager"  option.) 

In  the  second  they  received  the  unknown  turkey  for  sure.  Each  subject 
was  asked  to  make  a  choice  in  eleven  such  situations  with  the  probability 
varied  from  ten  out  of  ten  to  zero  out  of  ten  for  getting  the  brand  name 
turkey.  The  probability  at  which  the  subject  switched  to  the  unknown 
turkey  was  used  as  the  indifference  point.  The  hypothesis  that  people 
would  not  pay  more  for  a  brand  name  turkey  was  statistically  rejected; 
some  proportion  of  people  will  pay  more  for  brand  name  products. 

Finally,  the  method  has  been  researched  extensively. 107 


Constant  Probability  Method 


The  constant  probability  method  involves  wagers  where  the  probabili¬ 
ties  in  the  wager  are  held  constant  and  factor  levels  are  varied  until  an 
indifference  point  is  found.  The  method: 


P  1/2 
I 


3(4) 


c 

N 


uses  constant  probabilities 
depends  upon  indifference  judgments 
involves  three  or  four  factor  levels  at  a  time 
works  best  with  continuous  factors 
provides  direct  numerical  worth 


103  Hosteller,  F.  and  P.  Nogee.  An  experimental  measurement  of  utility. 
Journal  of  Political  Economics.  I95I,  59.  571-404. 

104  Raiffa,  H.  Preferences  for  multi-attributed  alternatives.  Rand  Corp. 
Memo  RM-5868-DOT-RC,  April  1968. 

106  Swalm,  R.  0.  Utility  theory--insights  into  risk  taking.  Harvard  Busi¬ 
ness  Review,  1966,  44,  125-156. 

106  Marquardt,  R.,  J.  Makens,  and  H.  Larzelere.  Measuring  the  utility 
added  by  branding  and  grading.  Journal  of  Marketing  Research,  1965, 

2j  45-50. 

107  Beach,  1972,  op.  cit.;  Edwards,  1954,  op.  cit.  ;  Slovic  and  Lichtenstein, 
1968,  op.  cit. 
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Process  Example.  Step  1:  Rank  the  factor  levels.  This  method  requires 
that  factor  level  and  preference  be  related.  If  they  are  not  related  some 
other  worth  assessment  method  must  be  used.  For  instance,  if  the  decision 
maker  prefers  a  middle  factor  level  to  both  extremes,  he  will  have  a  non¬ 
monotonic  worth  function  and  this  method  would  be  inappropriate.  If  the 
factors  are  continuous  and  monotonic,  only  the  most  and  least  preferred 
levels  need  be  specified.  As  an  example,  consider  the  assessment  of  worth 
for  horsepower.  The  range  of  acceptable  horsepower  has  been  set  at  50 
to  550  horsepower.  The  decision  maker  has  said  that  50  horsepower  is 
least  preferred  and  350  horsepower  is  the  most  preferred  level.  Assume 
that  the  worth  is  desired  on  a  0  to  100  scale.  Thus  we  assign  a  worth  of 
0'  to  50  horsepower  and  a  worth  of  100  to  350  horsepower. 

Step  2:  Construct  a  choice  wager  and  present  this  to  the  decision 
maker.  The  wager  is  set  up  with  the  most  and  least  preferred  factor 
levels,  each  having  a  probability  of  one-half  of  occurrence.  The  decision 
maker  is  asked  whether  he  prefers  the  wager  or  some  other  factor  level  for 
sure.  In  our  example  he  might  be  asked: 

"Would  you  prefer  to  accept  a  wager  where  the  proba¬ 
bility  of  getting  350  horsepower  is  one-half  and  the 
probability  of  getting  50  horsepower  is  one-half,  or 
would  you  prefer  to  have  200  horsepower  for  sure?" 

Step  3*  Vary  the  sure-thing  until  an  indifference  point  is  found. 

Assume  that  when  presented  the  above  choice  the  decision  maker  preferred 
the  wager.  In  order  to  make  the  sure-thing  more  attractive  it  is  changed 
from  200  to  25O  horsepower,  and  the  choice  is  presented  again. 

We  will  assume  the  decision  maker  is  indifferent  at  this  point.  We 
can  therefore  determine  a  worth  value  for  250  horsepower.  Because  we  are 
using  probabilities  of  one-half,  its  worth  will  be  midway  between  the 
worths  of  the  factor  levels  in  the  wagers;  thus  it  is  worth  50.  In  symbols 
this  is: 

W  (250  hp)  =  .5  W  (50hp)  +  .5  W  (550  hp)  =  .5  (0)  +  .5  (100) 

=  50  (4) 

Step  4:  Vary  the  wager.  We  construct  a  new  wager  using  two  of  the 
points  already  determined.  For  instance,  if  the  next  wager  involves 
25O  horsepower  and  350  horsepower,  we  find  the  worth  point  midway  between 
them.  Step  2  and  3  are  repeated  using  this  new  wager. 

As  points  are  found  they  could  be  plotted  so  as  to  estimate  the  worth 
curve  (Figure  16)# 
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Fiqure  16.  Worth  curve  estimated  for  horsepower 


Variations .  One  of  the  major  complaints  associated  with  the  gamble 
methods  is  that  one  option  involves  risk  while  the  other  is  a  sure-thing. 
For  instance,  if  the  decision  maker  is  averse  to  risk,  he  will  always 
choose  the  sure-thing.  To  alleviate  this  type  of  problem  we  can  have  the 
decision  maker  choose  between  two  wagers.  Although  this  is  more  difficult 
for  both  researcher  and  decision  maker,  response  bias  can  be  reduced  with 
proper  administration. 

The  method  described  in  steps  one  through  three  above  is  done  once  so 
that  three  points  are  known.  Using  these  three  points  we  can  ask  the 
decision  maker: 

"Would  you  prefer  a  wager  where  the  probability  of 
getting  25O  horsepower  is  .80  and  the  probability 
of  getting  50  horsepower  is  .20,  or; 

"Would  you  prefer  a  wager  where  the  probability  of 
getting  200  horsepower  is  .80  and  the  probability 
of  getting  55O  horsepower  is  .20,  or  are  you  indif¬ 
ferent  between  the  wagers?” 

The  unknown  factor  in  the  second  wager  is  varied  to  make  that  wager  more 
or  less  attractive.  In  this  case,  200  horsepower  would  be  changed  since 
we  have  worths  for  50,  25O  and  350  horsepower.  The  factor  is  varied 
until  an  indifference  point  is  found.  Assuming  that  the  decision  maker 
is  indifferent  to  the  choice  as  stated,  we  can  calculate  the  worth  of 
200  horsepower.  The  expected  worth  of  the  first  wager  is  known  to  be 
40  (0  x  .20  +  50  x  .80),  so  the  worth  of  200  is  found  by  solving  the 
following  for  W  (200),  read  ’’worth  of  200”: 

.80  x  W  (200)  +  .20  x  100  =  40 

W  (200)  =  20/. 80  =  25 

Thus,  the  worth  of  200  horsepower  is  25. 

A  point  above  25O  horsepower  can  be  found  by  setting  one  wager  as  25O 
horsepower  with  probability  of  .80  and  350  horsepower  with  .20,  and  setting 
the  second  as  200  horsepower  with  probability  of  .20  and  300  horsepower 
with  ^80.  The  300  horsepower  is  varied  to  find  the  indifference  point. 

The  process  is  repeated  until  enough  points  are  found  to  estimate  a  curve. 

Finally,  one  variation  involves  direct  magnitude  estimation  in  conjunc¬ 
tion  with  this  gamble  method.  The  decision  maker  estimates  directly 
the  value  of  the  sure-thing  that  makes  it  indifferent  to  the  wager.  For 
instance,  he  might  be  asked: 

"The  horsepower  of  the  automobile  you  will  purchase  will 
either  be  50  horsepower  with  a  probability  of  one-half,  or 
350  horsepower,  also  with  a  probability  of  one-half.  How 
many  horsepower  for  sure  would  make  you  indifferent  to  the 
wager?  That  is,  it  would  not  matter  to  you  whether  the 
horsepower  you  specify  is  used  for  sure?" 
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These  indifference  points  are  then  used  in  a  manner  similar  to  the 
basic  method.  This  variation  speeds  up  the  method  considerably. 


Characteristics .  Cautions:  All  of  the  cautions  of  the  variable 
probability  method  apply.  Some  of  the  problems  with  the  use  of  prob¬ 
abilities  are  reduced  with  this  method  because  probabilities  are  not 
varied,  and  generally  only  the  probability  of  one,-half  is  used  which  is 
familiar  to  most  people.  The  probabilities  of  course  must  add  to  one 
in  the  wagers,  and  the  situations  portrayed  should  be  as  realistic  as 
possible . 

Results:  The  result  of  this  method  is  numerical  worth  derived 
directly  from  the  indifference  points. 

Information  Needed:  The  specific  description  of  the  factor  being 
assessed  is  all  that  is  necessary.  However,  information  about  the  decision 
maker's  expertise  with  probabilities  would  be  helpful.  If  he  does  not 
understand  the  concept,  a  different  assessment  technique  should  be  used. 

Training  Required:  Same  as  variable  probability  method. 


Advantages .  The  advantages  of  this  method  are  the  same  as  with  the 
variable  probability  methods.  However,  this  method  may  reduce  biases 
caused  by  probability  preferences  since  all  wagers  contain  the  same 
probabilities.  Hence,  any  bias  will  be  constant  and  should  not  affect  the 
interval  worth  scale. 


Disadvantages.  The  problems  of  inveterate  gamblers  and  acceptability 
previously  described  in  the  variable  probability  method  are  also  dis¬ 
advantages  in  this  method. 


Checking  Techniques.  The  best  method  of  validation  involves  presenting 
the  decision  maker  with  redundant  choice  situations.  For  instance,  given 
the  t/orth  function  in  Figure  16,  we  could  ask  the  decision  maker: 

"Would  you  prefer  a  wager  where  the  probability  of  getting 
a  car  with  25O  horsepower  is  one-half,  and  for  a  car  with 
350  horsepower  is  also  one-half,  or  would  you  prefer  to 
have  a  car  with  300  horsepower  for  sure?" 

The  decision  maker  should  prefer  the  sure-thing,  since  the  expected 
worth  of  the  wager  is  75  anc*  the  worth  of  the  sure-thing  is  about  90 
(see  Figure  16). 
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Sources  of  Information.  The  theory  for  this  method  can  be  found  in 
Mosteller  and  Nogee  108  and  in  Raif fa, 109  who  used  the  method  in  trans¬ 
portation  system  studies.  The  method  is  described  in  Hurst  and  Siegel110 
and  Suppes  and  Walsh* * 111  although  different  techniques  were  used  to  obtain 
numerical  worth.  Royden,  Suppes  and  Walsh,112  Conrath  and  Deci,113  and 
Siegel114  have  used  a  variant  of  this  method.  The  method  and  several 
variations  are  described  by  Ackoff,  Gupta  and  Minas.116 

Finally,  the  variation  involving  direct  estimation  was  used  by 
Tversky116  in  a  study  of  additive  utility,  and  was  tested  by  Becker, 

DeGroot  and  Marschak.  117  They  presented  subjects  with  wagers  involving 
money  and  asked  them  for  the  lowest  amount  of  money  they  would  accept  in 
lieu  of  the  wager.  The  task  was  structured  so  that  it  was  in  the  subject's 
interest  to  give  the  cash  equivalent  of  the  wager  as  his  selling  price, 
and  this  price  was  used  as  the  indifference  point  in  the  method.  The  main 
finding  of  the  study  was  that  subjects  become  more  consistent  and  closer 
to  theoretical  behavior  with  practice.  This  supports  the  contention  that 
training  will  improve  the  gamble  methods. 


108  Mosteller  and  Nogee,  1951*  °P*  cit. 

109  Raif  fa,  1968,  op.  cit. 

110  Hurst  and  Siegel,  1956,  op.  cit. 

111  Suppes,  P.  and  K.  Walsh.  A  non-linear  model  for  the  experimental 

measurement  of  utility.  Behavioral  Science.  1959*  204-211. 

112  Royden,  H.,  P.  Suppes  and  K.  Walsh.  A  model  for  the  experimental 
measurement  of  the  utility  of  gambling.  Behavioral  Science.  1959> 

4,  11-18. 

113  Conrath,  D.  and  F.  Deci.  The  determination  and  scaling  of  a  bivariate 
utility  function.  Behavioral  Science.  I969,  14.  316-327* 

114  Siegel,  S.  A  method  for  obtaining  an  ordered  metric  scale.  Psycho- 
met  rika.  1959,  21,  207-216. 
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MULTIVARIATE  METHODS  OF  WORTH  ASSESSMENT 


The  multivariate  methods  are  those  which  are  specifically  designed 
to  consider  more  than  one  factor  at  a  time.  In  particular,  they  are 
well  suited  for  analyzing  dependent  factors,  although  they  are  not 
restricted  to  this  type.  When  developing  a  list  of  attributes  it  is 
often  impossible  to  obtain  a  set  of  attributes  which  is  both  exhaustive 
and  independent.  In  our  example  problem,  for  instance,  we  included  the 
attributes  Cost  of  Repairs  and  Reliability  which  are  valuewise  dependent. 
(The  attribute  Warranty,  which  is  probably  also  dependent  with  these, 
will  not  be  considered  in  this  discussion.) 

These  factors  should  not  be  assessed  as  separate  attributes;  rather, 
one  of  the  several  ways  of  handling  such  interdependent  factors  should 
be  used.  First,  all  possible  ( or  a  set  of  all  possible)  combinations 
of  the  factor  levels  could  each  be  assessed  pairwise  as  a  single  factor 
using  one  of  the  preceding  methods.  However,  for  factors  with  several 
levels,  the  number  of  possible  combinations  is  too  large  to  make  this 
strategy  feasible . 

Second,  the  attributes  could  be  combined  through  analysis  into  a  single 
attribute.  For  instance,  the  inverse  of  Reliability  can  be  thought  of 
as  number  of  breakdowns  per  year.  This  multiplied  by  Cost  of  Repairs 
gives  an  expected  repair  cost  per  year  which  could  be  used  instead  of  the 
two  separate  factors  of  Reliability  and  Cost  of  Repairs.  The  problem 
with  this  is  that  the  assumed  dependence  accounted  for  may  or  may  not 
reflect  the  dependence  seen  by  the  decision  maker.  However,  this  tech¬ 
nique  can  be  a  powerful  tool  in  obtaining  a  single  factor  from  a  set 
of  dependent  factors. 

Third,  the  two  dependent  factors  can  be  assessed  using  multivariate 
techniques.  This  strategy  allows  the  assessment  of  very  complex  depen¬ 
dencies  and  can  be  used  where  the  other  two  strategies  cannot.  The 
indifference  curve  or  trade-off  method  for  doing  this  will  be  described 
in  this  section.  Its  main  weak  point  is  that  it  is  a  long  and  tedious 
process.  However,  it  can  yield  much  information  on  complex  dependencies 
that  cannot  be  found  by  any  other  method. 


Indifference  Curve  Method 

The  indifference  curve  method  is  used  when  two  factors  are  involved 
and  their  joint  worth  is  to  be  assessed.  The  factors  can  be  dependent 
or  independent,  but  there  must  be  a  monotonic  relationship  between  worth 
and  factor  level  for  each  factor.  The  method  can  be  used  with  more  than 
two  factors  but  becomes  very  difficult.  It  Is  based  on  the  trade-off 
between  worths  of  continuous  factors  but  can  be  used  with  discrete 
factors  if  they  can  be  approximated  as  continuous. 
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N  -  uses  no  probabilities 

1  -  depends  upon  indifference  judgments 

2  -  involves  trwo  factor  levels  at  a  time 
C  -  works  best  with  continuous  factors 

R(N)  -  provides  ranking  of  factors  or  direct  numerical  worth 


Process  Example.  Step  1:  Identify  an  indifference  plane.  The  ini¬ 
tial  step  is  to  construct  a  graph,  or  plane,  such  that  the  axes  represent 
the  two  factors  under  study.  The  range  of  values  for  each  factor  should 
contain  all  logically  possible  factor  values. 

As  an  example  we  will  use  Cost  of  Repairs  and  Reliability  as  our 

factors.  They  can  be  assumed  to  be  dependent  since  high  cost  of  repairs 

with  high  reliability  might  be  as  preferred  as  low  cost  of  repairs  and 
low  reliability.  Figure  17  shows  the  indifference  plane  for  this  example, 
using  a  0  -  100  scale  for  each  factor. 

Step  2:  Determine  an  indifference  curve.  A  point  on  the  plane  is 
picked  at  random  to  become  a  reference  point  (for  example,  40$  reliable 
and  a  relative  repair  cost  of  30) .  The  point  should  be  in  a  part  of 

the  graph  which  seems  logical  to  the  decision  maker.  If  the  point  is  at 

an  extreme  (zero  cost,  zero  reliability,  for  instance)  he  may  have  trouble 
visualizing  what  it  means.  Other  points  on  the  indifference  plane  are 
then  compared  to  this  point.  The  researcher  asks  the  decision  maker  to 
state  a  preference  between  the  reference  point  and  a  comparison  point. 

For  instance,  in  our  example  he  might  be  asked: 

"Would  you  prefer  to  have  a  car  which  is  40$  reliable 
and  has  a  relative  repair  cost  of  30  or  would  you 
prefer  to  have  one  which  is  5 0$  reliable  and  has  a 
relative  repair  cost  of  or  are  you  indifferent 
between  the  two?” 

If  a  preference  is  expressed,  new  comparison  points  are  chosen  until 
he  is  indifferent  between  the  two.  The  two  points  then  lie  on  the  same 
indifference  curve;  i.e.,  have  the  same  worth.  This  procedure  is  con¬ 
tinued  until  sufficient  indifference  points  are  found  to  describe  a 
curve. 

There  are  several  techniques  for  choosing  new  comparison  points. 
Assume,  in  the  above  example,  that  worth  on  the  indifference  plane  in¬ 
creases  from  bottom  right  to  upper  left.  Then,  if  the  reference  point 
is  preferred,  the  new  comparison  point  should  be  to  the  left  and/or  above 
the  previous  comparison  point,  hence  increasing  its  worth.  Should  the 
comparison  point  be  preferred  to  the  reference  point,  the  new  comparison 
point  chosen  would  be  below  and/or  to  the  right  of  the  previous  one. 


-  55  - 


-K 


-R 


GO 

cr 

►H 

£ 

ft 

fe 

<3 


3 

a 


cQ 


K 


R 


bQ 


(%)  Aimwn^i 


CD 

CD 


-  56  - 


Figure  17.  Partially  completed  indifference  curve 


For  example,  assume  that  the  decision  maker  preferred  the  reference 
point  to  the  comparison  point,  but  was  indifferent  when  the  comparison 
point  was  (40,50),  The  indifference  plane  at  this  point  is  shown  in 
Figure  17* 

Once  an  indifference  point  is  found,  a  new  comparison  point  is  chosen 
so  as  to  define  more  of  the  indifference  curve.  The  next  point  to  check 
in  the  example  might  be  a  relative  cost  of  60  and  80$  reliability.  The 
objective  is  to  generate  enough  indifference  points  to  estimate  an  indif¬ 
ference  curve.  Assume  that  the  decision  maker  was  indifferent  between  the 
reference  point  (30  ,  40)  and  comparison  points  (10,30),  (40  ,  50),  (5^*60) 
and  (70*95)*  With  these  five  points  we  can  estimate  an  indifference  curve 
as  shown  in  Figure  l8. 

Step  3:  Choose  a  new  reference  point.  Once  an  indifference  curve 
has  been  determined  a  new  reference  point  is  chosen.  This  new  point  will 
form  the  basis  for  another  curve  on  the  plane.  Step  2  is  then  repeated 
for  each  new  reference  point  chosen.  The  final  outcome  of  this  step  is 
a  set  of  curves  which  cover  several  possible  combinations  of  the  two 
factors  (Figure  19)*  A  total  of  six  reference  points  were  chosen,  yielding 
the  six  curves  shown  in  Figure  19*  All  of  the  points  on  a  given  curve 
have  the  same  worth. 

Step  4:  Determine  the  worths  associated  with  the  curves.  Up  to 
this  point  we  have  determined  that  the  points  on  different  curves  have 
different  worths  but  we  have  not  determined  explicitly  how  much  more  one 
point  is  worth  than  another.  One  point  on  each  curve  is  chosen  from  which 
worths  can  be  assessed.  The  reference  points  make  good  choices,  since 
the  decision  maker  is  more  familiar  with  them  than  with  other  points  on 
the  curves.  The  worths  of  these  points  (six  in  our  example)  are  determined 
using  one  of  the  other  methods  described  in  this  paper.  This  step  will 
not  be  illustrated  here,  but  assume  that  the  worth  of  each  curve  is  as 
shown  in  Figure  19*  the  lowest  curve  being  worth  30  and  the  highest  worth 
80.  The  worth  of  any  point  on  the  plane  can  now  be  determined  by  interpo¬ 
lation,  if  we  assume  linearity  between  curves.  This  assumption  should  not 
introduce  any  serious  error,  but  should  be  used  with  caution. 


Variations .  It  is  possible  to  have  the  decision  maker  draw  indifference 
curves  directly.  Since  the  process  of  determining  curves  can  become  very 
time-consuming,  it  may  sometimes  be  advantageous  after  he  has  helped 
construct  one  or  two  curves  and  is  familiar  with  the  procedure  to  have 
him  draw  the  curves  directly  through  specified  reference  points.  He  must 
understand  what  the  curves  represent  and  how  they  are  derived  before 
this  direct  technique  is  attempted. 

One  variation  allows  direct  numerical  worth  of  independent  factors 
to  be  determined  directly  from  an  indifference  plane.  This  variation 
is  sometimes  called  the  trade-off  method.  Two  independent  factors  are 
assessed  using  the  indifference  method  described  above,  so  that  two 
indifference  curves  are  determined.  The  two  curves  should  be  close 
together  and  cover  as  much  of  the  factor  ranges  as  possible.  As  an 
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Figure  18.  Single  indifference  curve 
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example  assume  that  two  curves  have  been  found  on  the  Horsepower/ 

Miles  per  Gallon  indifference  plane.  The  two  curves  are  shown  in  the 
upper  right  corner  of  Figure  20.  A  "staircase"  is  drawn  between  the 
curves,  each  stair  or  rise  parallel  to  one  of  the  axes.  Since  the  worth 
of  each  curve  is  constant  over  its  length,  the  distances  between  "stairs" 
(shown  with  dotted  lines  to  the  axes)  represent  equal  intervals  of  worth. 
For  instance,  the  difference  in  worth  between  50  horsepower  and  200 
horsepower  is  the  same  as  the  difference  in  worth  between  200  horsepower 
and  240  horsepower.  Assuming  that  these  differences  are  worth  1.0,  we 
can  graph  the  functions  of  the  two  individual  factors  (Figure  20).  This 
method  allows  joint  assessment  of  two  independent  factors,  which  may  be 
meaningful  in  many  cases,  while  still  allowing  the  worths  of  individual 
factors  to  be  determined. 


Characteristics .  Cautions:  The  indifference  region  may  be  wide,  and 
it  may  be  hard  to  estimate  a  single  curve.  To  illustrate,  it  will  be 
assumed  that  a  decision  maker's  indifference  plane  (given  a  reference 
point)  can  be  divided  into  three  areas:  an  indifference  area,  a  reference- 
point-preferred  area  and  a  comparative-point-preferred  area.  Any  compar¬ 
ative  point  in  the  indifference  area  will  be  indifferent  in  terms  of  worth 
to  the  reference  point,  and  in  the  other  two  areas  either  the  comparative 
or  the  reference  point  will  be  preferred.  If  the  indifference  region  is 
wide  the  indifference  points  may  not  fall  into  a  smooth  curve;  in  Figure 
21  the  indifference  points  do  not  form  a  smooth  curve.  Although  the 
problem  will  seldom  be  as  severe  as  illustrated,  the  solution  is  the 
same.  A  curve  can  be  drawn  which  most  closely  follows  the  points  and 
passes  through  the  indifference  region. 

Equipment:  Graph  paper  and  pen  or  pencils  are  required. 

Results:  The  result  of  this  method  is  a  description  of  the  worth 

of  two  factors.  The  first  three  steps  will  provide  enough  information 
to  state  a  preference  relationship  between  any  two  factor  combinations. 

If  Step  4  is  completed,  a  quantitative  worth  value  can  be  estimated 
for  any  logical  combination  of  the  two  factors.  These  relationships 
will  depend  upon  any  interpolation  rules  that  are  used. 

Information  Needed:  Since  the  method  is  primarily  applicable  with 
two  factors,  the  range  and  precise  description  of  both  are  needed. 


Advantages .  Ability  to  assess  dependent  factors:  By  far  the  major 
advantage  of  this  method  is  the  fact  that  it  can  assess  the  worth  of 
factors  which  are  dependent.  While  other  methods  could  be  used  to  assess 
joint  worth,  this  method  is  clearly  the  most  straightforward  for  this 
purpose . 
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Figure  21.  Example  of  a  wide  indifference  region 


Disadvantages .  This  method  involves  a  long  tedious  procedure,  which 
may  tax  the  decision  maker.  The  variation  which  attempts  to  minimize 
this  problem  unfortunately  may  also  introduce  error  into  the  assessment. 

The  only  solution  to  the  problem  is  to  minimize  the  number  of  estimated 
points  per  curve  and  the  total  number  of  curves  in  the  plane.  There 
are  no  set  rules  to  follow  in  this  respect.  However,  complex  curves 
require  more  points  than  simple  relationships,  and  more  curves  make 
interpolation  easier.  In  addition,  as  the  researcher  becomes  more  famil¬ 
iar  with  the  technique  he  will  be  able  to  zero  in  on  difference  points  more 
rapidly,  so  practice  sessions  are  useful  before  attempting  a  final 
assessment  with  a  decision  maker. 


Checking  Techniques.  Determining  whether  the  indifference  curves 
are  logically  consistent  is  the  easiest  checking  technique.  The  indif¬ 
ference  curves  should  not  cross,  although  they  may  touch.  If  the  curves 
cross  this  implies  that  the  intersection  point  of  the  curves  is  indifferent 
to  points  in  four  different  directions,  which  implies  contradictory 
preferences.  Touching  is  permitted  because  even  though  the  curves  touch 
they  can  be  assumed  to  be  a  small  finite  distance  apart.  Touching  will 
most  often  occur  at  the  ends  of  the  indifference  curves.  The  curves  should 
also  be  checked  for  unusual  kinks  or  curves,  and  the  decision  maker  ques¬ 
tioned  about  any  irregularities. 

The  other  checking  technique  involves  presenting  the  decision  maker 
with  random  choice  preference  situations  after  (or  possibly  before)  the 
curves  have  been  completed.  In  our  example  (Figure  19)  he  may  be  asked 
whether  he  prefers  an  auto  which  has  a  relative  cost  of  repair  of  50  and 
30#  reliability  or  an  auto  which  has  a  relative  cost  of  repair  of  90  at 
60$  reliability.  He  should  prefer  the  former  (50,30).  If  he  prefers  the 
other,  the  discrepancy  should  be  cleared  up. 


Sources  of  Information.  This  method  of  worth  assessment  has  been 
documented  by  MacCrimmon  and  Toda118  and  Toda.  119  MacCrimmon  120  describes 


118  MacCrimmon,  K.  R.  and  M.  Toda.  The  experimental  determination  of 
indifference  curves.  Los  Angeles:  Western  Management  Science  Insti¬ 
tute,  Working  Paper  124,  1968. 

119  Toda,  M.  Indifference  map  method  for  estimating  utility  functions. 
Hokkaido  Report  of  Psychology,  HRP-1-71-12,  August  1971- 

120  MacCrimmon,  K.  R.  Improving  the  system  design  and  evaluation  process 
by  the  use  of  trade-off  information:  An  application  to  northeast 
corridor  transportation  planning.  Rand  Corp.  Memo  RM-5877”DOT, 

April  1969.  (PB  I85  166) 


-  63  - 


the  use  of  the  method  in  transportation  system  planning.  Slovic121  used 
the  technique  in  an  experiment  involving  trade-offs  between  cigarettes 
and  money.  The  trade-off  variation  is  described  by  Fishburn. 122 


SUMMARY 


Seven  general  methods  for  assessing  the  worth  of  decision  alternatives 
have  been  reviewed.  All  the  techniques  have  been  used  in  practice  and 
in  some  cases  compared  with  each  other.123'124  In  different  situations 
different  methods  have  been  found  to  be  most  effective,  and  no  single 
method  has  been  found  to  be  better  across  all  situations.  One  question, 
however,  remains:  Which  method  should  be  used  in  a  given  situation? 

Two  facts  form  a  loose  strategy  for  choosing  a  method:  the  method  must 
meet  the  requirements  of  the  situation,  and  it  must  meet  the  requirements 
of  the  decision  maker. 

First,  the  situational  requirements  must  be  considered.  The  researcher 
should  determine  the  type  of  output  that  is  needed.  For  instance,  it 
would  serve  no  function  to  determine  numerical  worth  if  only  a  ranking  of 
alternatives  is  required.  The  inputs  should  be  specified.  Are  they  dis¬ 
crete  or  continuous?  Can  they  logically  be  handled  independently?  For 
instance,  if  independence  is  justified  any  of  the  methods  are  applicable. 

If  not,  the  indifference  curve  method  or  other  multivariate  method  must 
be  used.  The  researcher  must  also  decide  whether  the  method  must  conform 
to  an  axiom  set,  or  whether  one  of  the  more  ad  hoc  procedures  will  suffice. 

Second,  the  method  chosen  should  be  tried  on  a  small  scale  in  order 
to  determine  if  it  is  acceptable  to  the  decision  maker.  Do  the  scenarios 
or  questions  seem  logical  and  realistic  for  the  situation?  Is  the  method 
comfortable  and  acceptable  for  both  the  researcher  and  decision  maker? 

Can  the  method  be  used  with  the  given  resources  of  time,  manpower,  etc.? 
Finally,  are  the  results  of  the  trial  reproducible?  Are  they  accurate? 

Although  there  will  usually  be  many  more  situational  requirements  to 
consider  or  questions  to  answer,  the  method  chosen  must  satisfy  the 
requirements  and  must  work  in  the  given  situation.  Satisfying  these  two 
criteria  may  involve  trade-offs  or  conceivably  may  require  that  a  method 
be  modified  in  order  to  find  the  best  possible  worth  assessment  method. 


121  Slovic,  P.  Consistency  of  choice  among  equally  valued  alternatives. 
Proceedings  of  the  76th  Annual  Convention  of  the  American  Psychological 

Association,  1968,  2,  57*59 • 

122  Fishburn,  P.  Independence,  trade-offs  and  transformations  in  bivariate 
utility  functions.  Management  Science,  1965,  11 ,  792-801. 

123  Eckenrode,  1965*  op,  cit. 

124  Bartlett,  C.,  F.  Heerman  and  S.  Rettig.  A  comparison  of  six  different 
scaling  techniques.  Journal  of  Social  Psychology,  I960,  51,  5^5*5 48. 
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Figure  22  presents  a  summary  comparison  of  the  methods  described  in 
this  paper.  No  attempt  is  made  to  describe  variations  of  methods  since 
the  same  characteristics  apply  to  each.  Detailed  discussion  of  the  entries 
in  the  matrix  can  be  found  in  the  appropriate  section  of  the  text. 
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TECHNIQUES 

RANKING  EQUIVALENCE  DIRECT  GAMBLE  INDIFFERENCE 

CHARACTERISTICS  .  METHODS  GROUPING  METHODS  METHODS  METHODS 
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Figure  22.  Comparison  of  worth  assessment  methods 
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